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EXECUTIVE  SUMMARY 

The  purpose  of  this  project  was  to: 

■  Systematically  investigate  the  effect  of  vegetation  cover  on  the  effectiveness  of  airborne 
lidar  in  munitions  management 

■  Examine  the  effects  of  lidar  point  classification  methods  on  lidar  data  density,  along  with 
implications  for  munitions  sites 

■  Summarize  the  sources  and  magnitudes  of  lidar  error,  along  with  implications  for 
munitions  sites 

■  Evaluate  the  ability  of  current  software  packages  to  automatically  identify  ground 
features  typical  of  munitions  sites 

-  Summarize  lessons  learned  from  the  Environmental  Security  Technology  Certification 
Program  (ESTCP)  Wide  Area  Assessment  (WAA)  Pilot  Program 

This  research  project  extends  the  earlier  work  with  light  detection  and  ranging  (lidar)  that  was 

carried  out  by  URS  Corporation  in  Project  200534.  Principal  findings  include  the  following: 

Activity  1  -  Systematically  investigate  vegetation  effects 

■  Measures  of  vegetation  height  and  density  derived  from  lidar  correlated  reasonably  well 
with  standard  measures  for  trees  but  not  for  grass. 

■  Vegetation  caused  a  decrease  in  lidar  density.  At  a  series  of  vegetated  test  plots  at  the 
Former  Camp  Beale  ESTCP  demonstration  site,  up  to  90%  of  laser  pulses  were 
determined  to  have  returned  from  vegetation.  Lidar  data  density  was  sufficient  to  model 
the  ground  surfaces  under  trees  at  this  site,  however  the  quality  of  the  surface  model  was 
degraded  under  brush. 

■  Lowering  the  lidar  data  density  at  test  plots  affected  feature  detection.  Lidar  data  at  the 
higher  densities  tested  (13.8  points  per  square  meter  [pts/m2])  detected  all  of  the  features 
at  the  test  plots,  as  verified  by  a  field  visit.  Significant  loss  of  feature  detection  began  to 
appear  between  6.9  pts/m2  and  3.4  pts/m2,  with  over  two  thirds  of  the  features  not  visible 
at  the  lowest  density  level  tested  (0.8  pt/m2). 

■  As  lidar  data  density  was  lowered,  ground  feature  detection  rates  declined,  however,  the 
effect  varied:  some  features  faded  gradually  as  lidar  density  was  lowered,  while  others 
did  not  disappear  at  all.  There  was  some  correlation  between  the  rate  of  degradation  and 
the  size  of  the  feature;  that  is,  larger  features  tended  to  disappear  more  slowly.  However, 
this  correlation  was  not  strong. 

■  The  implications  of  the  non-linear  relationship  between  lidar  data  density  and  feature 
detection  implies  that  there  is  no  threshold  lidar  density  at  which  no  features  will  be 
detected.  Rather,  lower-density  lidar  will  result  in  a  lower  degree  of  confidence. 

■  It  appears  that  increasing  lidar  density  will  have  a  point  of  diminishing  returns,  after 
which  additional  lidar  points  will  not  reveal  additional  features. 
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■  At  59  test  plots,  adding  all  of  the  lidar  points  within  40  cm  of  the  vendor’s  ground  surface 
revealed  additional  potential  features  at  36%  (21)  of  the  test  plots.  There  was  little  to  no 
negative  impact  to  the  quality  of  the  ground  surface  model,  even  though  this  procedure 
added  low  vegetation  and  noise.  This  implies  that  modifying  point  classification  methods 
to  include  more  points  in  the  ground  surface  model  can  potentially  improve  feature 
detection. 

■  Surface  model  cell  size  had  a  dramatic  impact  on  detection  of  small  features.  At  test 
plots  where  different  cell  sizes  were  tested,  using  smaller  cell  sizes  both  revealed 
additional  small  features  and  increased  the  clarity  of  the  features  shown.  It  appears  to  be 
appropriate  to  err  on  the  side  of  smaller  rather  than  larger  grid  cells. 

■  The  positional  accuracy  of  lidar  points  varies  with  many  factors,  with  the  largest 
instrument-based  factors  being  flight  height  and  distance  from  nadir  of  the  laser  signal. 
Lidar  acquisitions  at  the  ESTCP  demonstration  sites  were  conducted  from  300  to  1,000 
m,  where  instrument  error  is  relatively  low.  Lidar  collected  at  these  sites  met  their 
contracted  accuracy  specifications,  and  subsequent  acquisitions  should  successfully  meet 
similar  accuracy  specifications.  However,  where  accuracy  is  especially  important, 
contract  specifications  may  be  adjusted  to  require  the  vendor  to  fly  lower  and/or  use  a 
narrower  field  of  view. 

■  Positional  error  in  lidar  data  will  be  higher  on  sites  with  steep  terrain.  Where  appropriate, 
lidar  accuracy  can  be  assessed  independently  for  major  terrain  types. 

■  Surveyed  points  used  for  calibration  and  quality  control  of  lidar  must  be  from  a  source  of 
higher  quality  than  the  lidar  data  itself.  Survey  errors  in  control  and  calibration  points 
can  lead  to  incorrect  assessment  of  data  quality  or  errors  in  the  entire  lidar  data  set. 

■  Confidence  levels  for  feature  detection  using  lidar  can  be  mapped.  It  is  possible  to  map 
the  number  of  ground  returns  per  square  meter,  using  either  the  vendor’s  classification  or 
a  re-classified  data  set.  Site  managers  and  regulators  can  use  these  maps  to  classify  areas 
where  insufficient  lidar  points  reached  the  ground  surface  to  characterize  features  of  a 
given  size. 

■  The  results  of  this  investigation  point  out  the  importance  of  receiving  the  entire  lidar 
point  data  set  for  all  lidar  investigations.  Because  lidar  data  sets  are  very  large,  many 
vendors  only  deliver  derived  products  such  as  ground  surface  models.  By  requiring 
delivery  of  the  full  data  set,  Government  land  managers  can  evaluate  the  approaches  to 
point  classification  and  to  surface  model  creation  used  by  the  vendor,  and  make 
appropriate  adjustments. 

■  There  are  advantages  to  conducting  lidar  data  analysis,  including  creating  surface  models, 
in-house  by  Government  end-users  in-house,  rather  than  having  the  vendor  deliver  them. 
Creating  these  products  in-house  can  allow  for  experimentation  with  alternative  methods. 
However,  this  requires  that  Government  users  have  sufficient  software  tools  and  training 
to  accomplish  these  tasks.  Creating  DEMs  and  DTMs  in-house  can  be  challenging, 
especially  with  high-density  data  sets  where  the  number  of  points  is  very  large.  In  such 
cases  it  may  be  appropriate  for  end-users  to  work  with  sample  data  sets  to  determine,  in 
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consultation  with  vendors,  the  appropriate  specifications  for  analysis  products,  and  then 
request  that  the  vendor  create  and  deliver  the  final  products. 

Activity  2  -  Evaluate  current  software  packages 

Current  software  products  offer  many  useful  tools  for  manipulating  lidar  data,  but  of  those 
reviewed,  only  the  HTFC  software  by  Sandia  National  Laboratories  was  able  to  delineate  craters 
from  lidar-based  surfaces.  However,  software  for  manipulating  lidar  is  continuously  being 
developed,  and  software  vendors  should  be  contacted  periodically  to  review  new  developments. 
Government  offices  that  use  lidar  data  also  should  be  contacted  to  determine  which  software 
products  are  currently  in  use  and  whether  they  are  performing  well. 

Activity  3  -  Summarize  lessons  learned  from  the  ESTCP  WAA  Pilot  Program 

URS  and  US  ACE  delivered  a  revised  draft  guidance  document  summarizing  lessons  learned  to 
ESTCP  and  technical  peer  reviewers  in  July  2009.  This  document  was  revised  based  on  ESTCP 
and  peer  review,  and  is  submitted  as  Appendix  D. 

Cost  Benefit  Implications 

Cost  drivers  for  lidar  and  orthophoto  acquisition  are  discussed  in  detail  in  the  final  reports  for  the 
ESTCP  demonstration  sites  and  for  the  Pilot  Program  as  a  whole.  Additional  conclusions  from 
this  investigation  include: 

■  Cost  for  data  acquisition  at  vegetated  sites  should  not  be  significantly  higher  than  for 
non-vegetated  sites,  except  in  exceptional  circumstances.  Lidar  at  vegetated  sites  should 
be  collected  at  a  higher  density;  however,  with  the  introduction  of  higher-speed  sensors 
(up  to  250  kilohertz),  appropriate  data  densities  for  vegetated  sites  will  rarely  require 
additional  flight  lines. 

■  The  cost  of  orthophoto  acquisition  could  be  lower  at  vegetated  sites,  since  at  such  sites 
orthophotos  could  be  acquired  with  larger  pixel  sizes  or  omitted  in  favor  of  using  pre¬ 
existing  orthophotos. 

■  There  should  be  little  additional  cost  for  changing  data  classification  specifications  to 
return  more  points  to  the  ground  surface  model.  At  most,  the  vendor  would  charge  for 
some  initial  tests  of  alternate  processing  methods,  or  for  establishing  an  additional 
category  for  low  lidar  points.  These  additional  costs  should  not  be  high. 

■  This  project  showed  that  feature  detection  rates  did  not  go  down  substantially  between 
the  highest  data  density  tested  (13.8  pts/m2)  and  half  of  that  density  (6.9pts/m2).  This 
suggests  that  it  is  unnecessary  to  collect  extremely  high  lidar  data  densities.  This  may 
result  in  some  cost  savings. 
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1.0  INTRODUCTION 

Earlier  work  supported  by  the  Environmental  Security  Technology  Certification  Program 
(ESTCP)  has  shown  that  airborne  light  detection  and  ranging  (lidar)  can  be  a  useful  and  cost- 
effective  tool  in  the  assessment  and  characterization  of  munitions  response  sites.  The  purpose  of 
this  research  project  is  to: 

■  Systematically  investigate  the  effect  of  vegetation  cover  on  the  effectiveness  of  airborne 
lidar  in  munitions  management 

■  Examine  the  effects  of  lidar  point  classification  methods  on  lidar  data  density,  along  with 
implications  for  munitions  sites 

■  Summarize  the  sources  and  magnitudes  of  lidar  error,  along  with  implications  for 
munitions  sites 

■  Evaluate  the  ability  of  current  software  packages  to  automatically  identify  ground 
features  typical  of  munitions  sites 

■  Summarize  lessons  learned  from  the  ESTCP  Wide  Area  Assessment  (WAA)  Pilot 
Program 

1.1  BACKGROUND 

Many  millions  of  acres  of  current  and  former  Department  of  Defense  (DoD)  lands  are  potentially 
contaminated  with  military  munitions  or  their  components.  On  the  majority  of  these  sites, 
munitions  are  concentrated  in  specific  ranges  and  training  areas,  while  the  remainder  of  the  site 
is  ordnance-free.  Locating  the  site  of  contamination  can  be  difficult,  in  part  because  historical 
records  are  often  incomplete  or  inaccurate.  The  cost  of  traditional  surveys  using  magnetometry 
and  electromagnetic  induction  (EMI)  can  be  very  high,  and  this  has  driven  the  search  for 
innovative  methods  to  reduce  costs. 

This  research  project  is  an  outgrowth  of  the  WAA  approach  to  site  characterization  and  the 
ESTCP  WAA  Pilot  Program.  WAA  is  a  procedure  used  to  gather  “a  preponderance  of  evidence 
that  improves  the  understanding  of  the  site  and  builds  confidence  in  the  conclusions.  A  suite  of 
commercially  available  technologies  provides  data  to: 

■  Identify  areas  of  concentrated  munitions  use 

■  Collect  information  that  will  support  decisions  on  areas  with  no  indication  of  munitions 
presence 

■  Collect  data  to  support  planning,  prioritization  and  contracting  when  a  site  ultimately 
must  be  cleaned  up  ”  (Nelson  et  al.  2008) 

Previous  work  by  ESTCP  demonstrated  that  lidar  could  be  an  effective  addition  to  this 
combination,  helping  to  detect  and  delineate  munitions  response  sites  (MRS),  correcting  the 
initial  conceptual  site  model  (CSM),  and  providing  data  in  support  of  subsequent  investigation. 
Results  from  lidar  and  orthophotos  provided  cross-validation  of  magnetometry  and  EMI  data, 
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leading  to  higher  confidence  levels,  and  the  combination  of  technologies  employed  in  the  pilot 
program  was  found  to  be  cost-effective  (Nelson  et  al.  2008). 

The  current  demonstration  extends  ESTCP’s  earlier  work  with  lidar  that  was  carried  out  by  URS 
Corporation  in  Project  200534. 

1.2  OBJECTIVES  OF  THE  DEMONSTRATION 

This  demonstration  is  an  extension  of  the  WAA  Pilot  Program,  investigating  and  providing 
additional  guidance  to  Government  land  managers  on  the  appropriate  use  of  lidar  and 
orthophotography  at  munitions  sites. 

Activity  1:  Systematically  investigate  vegetation  effects  on  airborne  lidar  in  munitions 
management 

The  objectives  for  this  activity  were  to: 

a.  Examine  the  performance  of  lidar  and  orthophotography  for  detection  of  MRS  and 
munitions  and  explosives  of  concern  (MEC)-related  features  under  a  range  of  described 
vegetation  conditions 

b.  Investigate  the  relationship  between  vegetation  density  and  lidar  point  density,  and  its 
relationship  to  object  detection 

c.  Provide  guidance  to  Government  land  managers  on  the  effects  of  vegetation  conditions 
on  the  size,  shape,  and  location  of  features  that  can  be  detected  with  lidar  and 
orthophotography 

d.  Investigate  the  impact  of  lidar  point  classification  methods  on  ground  surface  models  and 
feature  detection  and  determine  whether  changes  to  point  classification  methods  would 
result  in  increased  detection  capability  (this  objective  was  added  following  the  In- 
Progress  Review  meeting  of  February  2008) 

e.  Summarize  the  types  and  magnitudes  of  lidar  error  and  their  implications  for  the  use  of 
lidar  at  military  munitions  sites  (this  objective  was  added  following  the  In-Progress 
Review  meeting  of  October  2008) 

Activity  2:  Evaluate  current  software  packages  for  their  ability  to  automatically  identify 
ground  features  typical  of  munitions  sites 

The  objective  for  this  activity  was  to: 

a.  Evaluate  the  ability  of  commercial,  off-the-shelf  software  to  identify  MEC-related  ground 
features  from  lidar  data  sets  typical  of  munitions  sites.  Analyze  features  including  craters 
at  a  variety  of  sizes  from  2-5  meters  (m)  in  diameter,  along  with  bombing  targets  and 
other  objects  constructed  from  berms. 
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Activity  3:  Summarize  lessons  learned  from  the  ESTCP  WAA  Pilot  Program 
The  objectives  for  this  activity  were  to: 

a.  Accurately  summarize  the  lessons  learned  during  previous  lidar  and  orthophotography 
testing  at  the  ESTCP  WAA  demonstration  sites,  as  well  as  the  results  of  Activities  1  and 
2,  in  a  single  guidance  document 

b.  Provide  peer  review  of  the  draft  guidance  document 

c.  Contribute  useable  and  meaningful  guidance  to  Government  land  managers  considering 
the  acquisition  and  use  of  lidar  and  orthophotography  data  in  munitions  management 

1.3  REGULATORY  DRIVERS 

MEC  remediation  is  generally  conducted  under  the  authority  of  the  Comprehensive 
Environmental  Response,  Compensation,  and  Liability  Act  (CERCLA).  With  many  millions  of 
acres  of  land  potentially  contaminated,  estimates  of  the  cost  of  elimination  of  environmental 
liability  under  this  statute  at  known  and  former  DoD  sites  range  as  high  as  several  hundred 
billion  dollars.  These  potentially  high  costs  have  led  to  interest  in  developing  innovative 
investigative  or  screening  methods  such  as  WAA,  in  order  to  reduce  the  costs  of  associated 
remediation  activities. 
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2.0  TECHNOLOGY 

This  section  provides  an  overview  of  lidar  and  orthophotography  technology. 

2.1  TECHNOLOGY  DESCRIPTION 

Lidar  technology  uses  an  airborne  laser  and  sensor  to  map  ground  and  non-ground  surfaces.  The 
laser  and  sensor  are  mounted  in  a  helicopter  or  small  aircraft,  and  Global  Positioning  System 
(GPS)  and  inertial  measurement  unit  (IMU)  equipment  is  used  to  locate  the  sensor  in  the  air. 

The  time  of  return  of  the  laser  signal  allows  for  the  accurate  calculation  of  its  point  of  reflection 
from  the  ground,  buildings,  or  vegetation  (Figure  1).  Each  laser  return  is  classified  as  returning 
from  the  ground  surface,  vegetation,  structures,  or  other  objects. 

Figure  1 :  Lidar  System  Operations 

Lidar  points  have  a  vertical  positional  accuracy  of 
approximately  15  centimeters  (cm)  on  soft  ground  surfaces 
and  a  horizontal  accuracy  of  approximately  60  cm,  both 
compared  to  surveyed  control  points.  Actual  point-to-point 
variation  is  typically  less  than  15  cm.  Accuracy  is  greater  on 
relatively  flatter  and  harder  surfaces. 

Once  elevation  data  are  collected  in  the  form  of  lidar  points, 
surface  models  are  created  and  analyzed,  and  maps  and 
analysis  products  are  created.  The  surface  modeling  process 
can  be  conducted  using  standard  Geographic  Information 
System  (GIS)  software  and  methods,  and  much  of  the 
process  can  be  automated. 

Lidar  has  been  become  a  standard  tool  for  many  applications 
that  require  detailed  ground  surface  models.  Its  popularity  is  due  to  its  ability  to  provide  more 
accurate  topographic  data  than  other  sources,  at  a  reasonable  cost  compared  to  alternatives  such 
as  ground  survey  or  photogrammetry. 

An  orthophotograph  is  a  digital  aerial  photograph  that  has  been  geometrically  corrected  for 
topographic  relief,  lens  distortion,  and  camera  tilt.  Individual  digital  images  are  assembled  into  a 
single  mosaic,  which  is  then  corrected  using  sophisticated  mathematical  methods  to  permit  the 
accurate  spatial  location  of  each  photo  pixel.  This  process,  called  orthorectification,  allows  the 
images  to  be  used  in  a  GIS  or  computer-aided  drafting  and  design  (CADD)  system  with  other 
spatial  data,  such  as  contour  lines  or  survey  data  (Nelson  et  al.  2008).  While  orthophotos  are 
adjusted  so  that  the  pixels  are  spatially  accurate,  this  only  encompasses  pixels  representing  the 
ground  surface.  Pixels  representing  the  tops  of  buildings  or  tall  trees  can  be  slightly  displaced 
(Appendix  B). 

Digital  orthophotography  has  been  available  commercially  since  the  early  1980s,  with  steady 
improvement  in  the  resolution  (i.e.,  pixel  size)  and  precision  (i.e.,  pixel  placement)  of  the  images 
as  the  technology  of  digital  cameras,  GPS,  and  IMU  systems  has  advanced.  Digital  cameras 
have  largely  replaced  film  cameras  in  the  productions  of  orthophotos  (Dold  2008). 
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Airborne  digital  cameras  have  been  integrated  with  lidar  sensors  successfully  and,  because  the 
two  sensors  use  the  same  GPS  and  Inertial  Navigation  System  (INS),  the  two  data  sets  can  be 
integrated  very  accurately.  Vendors  generally  guarantee  a  horizontal  accuracy  of  three  pixel 
widths  compared  to  ground  control  for  orthophotography,  and  spatial  integration  of  orthophotos 
and  lidar  within  two  pixel  widths. 

Final  orthophoto  pixel  size  depends  on  the  flight  altitude  and  the  camera  specifications.  For 
munitions  sites,  orthophoto  pixel  size  is  typically  from  0.3  m  (about  1  foot)  down  to  0. 1  m  (about 
4  inches)  (Nelson  et  al.  2008).  Pixel  sizes  smaller  than  this  generally  are  impractical  because  of 
the  low  flight  elevations  and  slow  flight  speeds  required  to  collect  properly  overlapping  images, 
and  the  large  numbers  of  images  that  would  need  to  be  combined  into  the  image  mosaic.  Many 
areas  also  have  existing  orthophotography  available,  with  pixels  in  the  0. 5-1.0  m  size  range.1 

A  more  complete  description  of  lidar  and  orthophotography  technology,  including  a  discussion 
of  system  components,  multi-return  capabilities,  and  other  characteristics,  is  provided  in  the 
ESTCP  White  Papers  Errors  in  Lidar  Data:  Implications  for  Investigation  of  Military  Munitions 
Sites  (Appendix  B)  and  Effects  of  Lidar  Point  Classification  Methods  on  Surface  Model  Creation 
and  Feature  Identification  (Appendix  C). 

2.2  TECHNOLOGY  DEVELOPMENT 

Between  2005  and  2007,  ESTCP  conducted  a  pilot  program  to  test  the  effectiveness  of  a  multi¬ 
technology  approach  to  unexploded  ordnance  (UXO)/MEC  WAA. 

The  first  phase  of  the  ESTCP  program  was  carried  out  at  seven  sites.  The  first  four  were  desert 
sites  where  the  technologies  were  expected  to  perform  well.  The  last  three  were  sites  with  more 
challenging  conditions  and  were  chosen  to  test  the  limits  of  the  various  technologies  chosen. 

The  WAA  Pilot  Program  examined  the  use  of  lidar,  orthophotography,  helicopter  magnetometry, 
towed-array  magnetometry,  synthetic  aperture  radar,  hyperspectral  sensing,  and  statistically- 
based  transect  design. 

At  the  five  sites  where  lidar  and  orthophotos  were  tested,  the  results  were  positive.  Lidar  and 
orthophotos  were  used  to  locate  berms,  bombing  targets,  and  individual  craters,  even  where  these 
features  were  highly  eroded  and  could  not  be  detected  by  ground-based  field  crews.  The  location 
of  these  features  was  used  to  correct  the  initial  CSM,  to  refine  the  boundaries  of  MRS,  and  to 
support  and  help  direct  the  use  of  subsequent  technologies  that  directly  detect  ordnance. 

Example  results  are  shown  in  Figure  2. 


1  States,  cities  and  counties  acquire  orthophotos  for  a  variety  of  purposes,  not  generally  including  detection  of  smaller 
objects.  Consequently,  these  entities  generally  do  not  incur  the  considerably  larger  costs  of  higher-resolution 
orthophotos.  These  coarser  images  can  nevertheless  be  useful  on  munitions  sites  with  tree  cover,  where  individual 
features  are  not  visible  and  the  primary  use  of  the  images  will  be  overall  mission  planning. 
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Figure  2:  ESTCP  Demonstration  Sites,  Example  Munitions-Related  Features 


Kirtland  Air  Force  Base  Precision  Bombing  Range  demonstration  site.  Bombing  targets  from  the  1940s  are  visible  in 
lidar-derived  surface  models.  These  targets  are  highly  eroded  and  cannot  be  seen  from  the  ground  or  in  aerial  photos. 


Kirtland  demonstration  site.  Bombing  target  cross-hairs 
visible  in  10-cm  pixel  orthophoto. 


Victorville  Demolition  Bombing  Range  demonstration  site. 
Potential  crater  locations  outside  of  the  original  target  area 
boundary. 
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Former  Camp  Beale  demonstration  site,  bombing  target 
(red  circle)  located  outside  of  mapped  target  locations 
based  on  historical  records. 


Pueblo  Precision  Bombing  Range  demonstration  site,  crater 
density  map  used  to  refine  original  MRS  boundary.  Source: 
ESTCP  and  Sky  Research. 


2.3  ADVANTAGES  AND  LIMITATIONS  OF  THE  TECHNOLOGY 

Lidar  and  orthophotography  have  the  advantages  and  limitations  discussed  in  Sections  2.3.1  and 
2.3.2  compared  to  the  current  approach  to  creating  CSMs  through  reviewing  historic  documents, 
ground-based  geophysical  surveys,  and  statistical  modeling.  These  advantages  and  limitations 
were  reported  in  URS’s  final  reports  for  the  ESTCP  demonstration  sites  it  examined  (URS  2007 
and  2008a)  and  for  the  overall  program  (Nelson  et  al.  2008),  and  are  consistent  with  the  findings 
of  this  demonstration. 

2.3.1  Advantages  of  the  Technology 

Using  lidar  and  orthophotos  offers  several  advantages  compared  to  the  traditional  approaches  to 
site  investigation: 

■  Rate  of  coverage.  In  an  operational  setting,  data  collection  rates  of  5,000  acres  per  day 
or  higher  can  be  expected  for  lidar  and  orthophotos.  This  compares  favorably  to  typical 
collection  rates  of  around  500  acres  per  day  for  helicopter-based  magnetometry,  5  to  20 
acres  per  day  for  towed-array  magnetometry,  and  1  to  5  acres  per  day  for  man-portable 
magnetometry. 

■  Enhanced  planning  and  risk  assessment.  Because  they  can  cover  entire  sites  relatively 
quickly  and  at  lower  cost,  these  technologies  can  be  used  to  locate  and  prioritize  the  use 
of  more  costly  low-altitude  or  ground-based  technologies. 
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■  Ability  to  delineate  MRS  and  MEC-related  features.  Lidar  and  orthophotography  can, 
under  some  circumstances,  successfully  reveal  MRS  and  MEC-related  surface  features 
even  many  years  after  their  last  use. 

■  Cross-validation.  Lidar  and  orthophotos  can  cross-validate  the  results  of  other 
technologies,  leading  to  enhanced  confidence  in  results. 

■  Other  benefits.  Both  technologies  provide  highly  detailed  topographic  data  that  can  be 
integrated  into  a  facility’s  CADD  or  GIS  system  and  used  in  subsequent  phases  of  site 
investigation,  site  remediation,  and  range  management. 

2.3.2  Limitations  of  the  Technology 

The  primary  limitation  of  lidar  and  orthophotography  is  that  neither  technology  can  directly 
detect  ordnance  or  MEC  components,  such  as  ferrous  scrap.  Consequently,  lidar  and 
orthophotos  are  not  substitutes  for  technologies  that  directly  detect  ordnance,  such  as 
magnetometry  or  EMI.  Rather,  lidar  and  orthophotos  are  used  to  give  a  rapid  assessment  of 
potential  areas  of  ordnance  use,  based  on  ground  features  such  as  target  objects,  crater  fields, 
open  bum/open  demolition  (OB/OD)  areas,  berms,  or  roads.  Areas  appearing  to  be  free  of 
evidence  of  munitions  use  in  lidar  and  orthophotos  still  need  to  be  surveyed  with  magnetometry 
or  EMI;  however,  the  initial  level  of  effort  with  these  technologies  could  potentially  be  reduced, 
and  then  adjusted  depending  on  the  initial  results. 

Lidar  and  orthophotos  are  best  used  as  the  first  step  in  site  investigation  to  quickly  identify  areas 
of  potential  contamination  and  other  areas  of  interest  and  to  help  direct  the  application  of 
subsequent  technologies.  Lidar  and  orthophotos  will  be  less  useful  on  sites  that  already  have 
been  well  characterized  using  other  methods,  or  on  small  sites  that  can  be  surveyed  quickly  using 
ground  crews. 

An  additional  limitation  of  orthophotos  is  their  limited  usefulness  at  highly  vegetated  sites.  At 
such  sites,  orthophotos  show  only  the  tops  of  the  vegetation,  and  will  have  limited  usefulness. 
Lidar  will  provide  useable  surface  models  through  many  vegetation  conditions,  and  will  show 
severe  degradation  only  where  the  vegetation  is  particularly  dense.  However,  vegetation  will 
lower  the  density  of  lidar  returns  from  the  ground  surface  and  will  thus  lower  confidence  levels 
for  detecting  features  under  vegetation. 
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3.0  PERFORMANCE  OBJECTIVES 

URS’s  ESTCP  Draft  Demonstration  Plan  (URS  2008b)  establishes  performance  objectives  for 
the  current  demonstration.  Table  1  lists  these  objectives  plus  additions  related  to  white  papers 
requested  by  ESTCP  following  the  February  and  October  2008  In-Progress  Review  meetings. 


Table  1:  Performance  Objectives 


Performance 

Objective 

Metric 

Action  to  Achieve 
Metric 

Sampling 
Frequency  or 
Timing 

Desired  Result 

Activity  1:  Systematically  Investigate  Vegetation  Effects  -  Field  Data  Collection 

Delineate  vegetation 
classes  using  photos 
and  lidar  data 

All  major  vegetation 
classes  identified 

Delineate  from 
orthophotos  followed  by 
QA/QC  review  by 
qualified  staff 

Once  at  start  of 
project 

1 00%  identification 
of  all  major 
vegetation  classes 

Calculate  lidar 
ground  and 
vegetation  point 
percentages 

Percentages 
accurately  calculated 
on  a  per  square  meter 
basis  using  a  3  m  cell 
size  to  show  variation 
across  the  study  area 

Use  of  standard  GIS 
calculation  methods 
followed  by  QA/QC 
review 

Once  at  start  of 
project 

100%  of  study  area 
characterized 

Calculate  lidar  data 
density  variability 

Density  of  ground 
lidar  points  and  all 
lidar  points  accurately 
calculated  on  a  per- 
meter  basis  to  show 
variation  across  the 
study  area 

Use  of  standard  GIS 
calculation  methods 
followed  by  QA/QC 
review 

Once  at  start  of 
project 

1 00%  of  study  area 
characterized 

Select  representative 
field  plots 

Representative  study 
areas  selected 

Select  at  least  50  plot 
locations,  with  plots 
distributed  in  all 
vegetation  classes 

Once  at  start  of 
project 

Plots  placed  in  all 
vegetation  classes 

Collect  field 
vegetation  density 
data 

Collect  accurate  data 
on  vegetation 
conditions  at  each  plot 

Use  standard  forestry 
methods  to  determine 
vegetation  density 

Once 

All  field  data 
collected  to  follow 
SOP 

Collect  field  ground 
feature  data 

Collect  accurate  data 
on  ground  features  at 
each  plot 

Incorporate  field 
procedures  into  SOP, 
followed  by  QA/QC 
review 

Once 

All  field  data  to 
follow  SOP 

Activity  1:  Systematically  Investigate  Vegetation  Effects  -  Data  Analysis 

Step  1:  Classify  Vegetation 

Lidar  ground  point 
density  vs.  field 
vegetation  density 

Difference  between 
vegetation  density  as 
measured  by  lidar  and 
field  methods 

Standard  GIS  analysis 
methods,  followed  by 
QA/QC  review 

Once 

Correlation  should 
be  highest  in  plots 
with  a  single,  simple 
vegetation  class 
such  as  grass  or 
trees 
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Performance 

Objective 

Metric 

Action  to  Achieve 
Metric 

Sampling 
Frequency  or 
Timing 

Desired  Result 

Lidar  vegetation 
heights  vs.  field 
vegetation  heights 

Difference  between 
vegetation  heights  as 
measured  by  lidar  and 
field  methods 

Comparison  of  lidar  and 
field  data 

Once  -  these 
may  vary 
seasonally  (for 
grasses) 
depending  on 
the  timing  of 
the  lidar  survey 
and  site  visit) 

Correlation  should 
be  within  5  feet  for 
isolated  trees 

Step  2:  Identify  Ground  Features 

Field  ID  of  features 
vs.  ID  of  features 
from  lidar 

Percentage  detection 
and  false  alarm  rate 
for  features  detected 
using  lidar  vs.  field 
observation 

Visual  observation  of 
lidar  data  followed  by 
comparison  of  results  of 
lidar  and  field 
observations 

Once 

Correlation  should 
be  100%  for 
features  over  1  m  in 
size  in  plots  with  no 
covering  vegetation 

Effects  of  lidar  point 
density  on  feature 
detection 

Number  of  features 
visible  at  successively 
lower  point  densities 

Artificially  lower  point 
density  through  a  series  of 
levels,  create  surface 
models  and  examine  the 
detection  of  features 
visible  at  the  highest 
density 

Once 

Determine  the  lidar 
data  density  where 
detection  falls  off 
substantially 

Step  3:  Produce  White  Paper  on  Lidar  Point  Classification  Methods 

Determine  the 
percentage  of  lidar 
points  misclassified 
as  non-ground 
returns 

Percentages  of  points 
classified  as  ground  or 
non-ground  in 
selected  vegetation- 
free  locations 

Analyze  in  GIS  to 
determine  percentage 
values  at  selected  test 

areas 

Once 

Determine 
magnitude  of 
misclassification 
problem 

Determine  the 
source  of 

misclassification  of 
lidar  points  as  non¬ 
ground  returns 

Clear  understanding 
of  the  source  of 
misclassification 

Converse  with  vendors’ 
data  classification  staff 
and  examine  software 
manuals  for  classification 
software 

Once 

Clear  understanding 
of  the  source  of 
misclassification 

Examine  the 
potential  benefit  of 
reclassification  of 
lidar  points 

Approximate  number 
and  size  of  features 
visible  after 
reclassification 

Create  new  surface 
models  using  reclassified 
points  and  examine  to 
determine  whether 
additional  features  are 
visible 

Once 

Estimate  of  the 
potential  benefit  of 
point 

reclassification 

White  paper 
technical  accuracy 

White  paper  is 
technically  accurate 

Peer  review  by  specialists 
from  industry  and 
universities 

Once 

Document  will  be 
technically  accurate 

White  paper 
readability 

Document  is  well 
written  and 
understandable  by 
non-specialists 

Technical  editing  and 
QA/QC  review 

Once 

Document  will  be 
understandable 
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Performance 

Objective 

Metric 

Action  to  Achieve 
Metric 

Sampling 
Frequency  or 
Timing 

Desired  Result 

Step  4:  Produce  White  Paper  on  Lidar  Error 

White  paper 
technical  accuracy 

White  paper  is 
technically  accurate 

Peer  review  by  specialists 
from  industry  and 
universities 

Once 

Document  will  be 
technically  accurate 

White  paper 
readability 

White  paper  is  well 
written  and 
understandable  by 
non-specialists 

Technical  editing  and 
QA/QC  review 

Once 

Document  will  be 
understandable 

Activity  2:  Evaluate  Current  Software  Packages 

Prepare  appropriate 
lidar  data  sets  for 
subsequent  testing 

Accurate  ID  of  ground 
features  for  testing 

Visual  inspection 
followed  by  QA/QC 
review 

Once  at  start  of 
project 

100%  confidence  in 
test  features  to  be 
given  to  vendors 

ID  of  features  using 
automated  methods 

Percent  detection  and 
false  alarm  rate  of 
automated  methods 
vs.  visual  inspection 
results 

Compare  results  of  visual 
inspection  and  automated 
methods 

Once  from  each 
vendor 

Percent  detection 
will  be  highest  for 
isolated,  well- 
defined  features. 

False  alarm  rate  will 
be  lowest  for 
relatively  flat, 
smooth  ground 
surfaces. 

Activity  3:  Summarize  Lessons  Learned  from  the  ESTCP  WAA  Pilot  Program 

Usability 

Guidance  document 
applicable  to  the 
requirements  of 
present  and 
anticipated  munitions 
management 
programs 

Peer  review  by  DoD 
program  and  policy  staff 

Once 

Document  will  be 
useable 

Technical  accuracy 

Guidance  document  is 
technically  accurate 

Peer  review  by  specialists 
from  industry  and 
universities 

Once 

Document  will  be 
technically  accurate 

Readability 

Guidance  document  is 
well  written  and 
understandable  by 
non-specialists 

Technical  editing  and 
QA/QC  review 

Once 

Document  will  be 
understandable 

DoD  -  Department  of  Defense 
GIS  -  Geographic  Information  Systems 
ID  -  identification 
m  -  meter 

QA/QC  -  quality  assurance/quality  control 
SOP  -  standard  operating  procedure 
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4.0  SITE  DESCRIPTION 

No  new  sites  were  developed  for  this  demonstration.  Instead,  lidar  data  from  four  ESTCP 
demonstration  sites  where  lidar  was  collected  previously  were  re-examined.  These 
demonstration  sites  are  described  in  the  Final  Report  for  the  WAA  Demonstration  Project 
(Nelson  et  al.  2008).  A  summary  of  site  characteristics  is  provided  in  Table  2. 


Table  2:  ESTCP  Demonstration  Sites 


Name 

Pueblo  PBR 

Kirtland  AFB  PBR 

Victorville  DBR 
“Y”  and  PBR  15 

Former  Camp 
Beale 

Location 

Pueblo,  CO 

Albuquerque,  NM 

Victorville,  CA 

Marysville,  CA 

Size  (acres) 

6,710 

5,000 

5,640 

87,672 

Date  of 
acquisition 

August  20-23, 
2004 

August  9-11,  2005 

January  24-25,  2006 
February  3-4,  2006 

July  22-26,  2006 

Lidar  vendor 

Sky  Research 

Terra  Remote  Sensing 

Terra  Remote 
Sensing 

Terra  Remote 
Sensing 

Lidar  data  sets 
(#> 

1 

4 

2 

2 

Overall  lidar 
data  density 
(pts/m2) 

5. 1 6a 

900  m  North  Block:  1 .4 

900  m  South  Block:  1.6 

450  m  north  block:  5.2 

450  m  south  block:  4.1 

300  m  east- west  north  block: 
5.2 

300  m  north-south  north 
block:  6.5 

300  m  flight  1  south  block:  5.9 
300  m  flight  2  south  block:  6.1 

450  m  flight:  4.8 

300  m  flight:  6.4 

450  m  flight:  13.8 
300  m  flight:  13.7 

Orthophoto  pixel 
size  (cm) 

18 

12/20 

10 

10 

Reference 

Sky  Research 
2008 

URS  2007 

URS  2007 

URS  2008a 

a.  Sky  Research  reported  density  as  average  point  separation  rather  than  average  points  per  square  meter.  The  reported  point  spacing  of  0.44  m 
was  converted  to  average  points  per  square  meter  using  the  formula  D  =  l/(sa)2  where  D  is  the  density  in  points  per  square  meter  and  sa  is  the 
average  separation  in  meters. 

AFB  -  Air  Force  Base 
DBR  -  demolition  bombing  range 
PBR  -  precision  bombing  range 
pts/m2  -  points  per  square  meter 
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4.1  SITE  SELECTION 

4.1.1  Activity  1:  Systematically  Investigate  Vegetation  Effects 

For  Activity  1,  URS  used  lidar  and  orthophotography  data  from  a  study  area  within  the  Former 
Camp  Beale  demonstration  site  in  the  Activity  1  analysis.  This  data  set  was  chosen  based  on  the 
following  criteria: 

■  The  study  area  contains  several  vegetation  types,  but  vegetation  is  not  too  complex  for  an 
initial  study  of  this  type. 

■  The  study  area  is  located  on  public  land  within  the  Spenceville  Wildlife  Refuge.  Because 
the  site  is  public  property  with  a  single  manager,  site  access  was  easier  than  with  the 
multiple  private  property  owners  in  the  remainder  of  the  Former  Camp  Beale  site. 

■  The  lidar  data  for  the  study  area  is  very  dense,  and  provides  a  useful  “best  case”  test  data 
set. 

■  Approximately  50  features  of  interest  within  the  study  area  had  been  visited  by  field 
crews  already.  Data  for  these  features  included  GPS  location  and  a  site  photo. 

■  Because  the  study  area  is  managed  as  a  wildlife  refuge,  there  had  been  no  vegetation 
clearing  since  the  lidar  data  was  collected.  Given  the  dry  and  undisturbed  nature  of  the 
study  area,  vegetation  conditions  were  likely  to  be  relatively  unchanged  between  lidar 
data  collection  in  June  2006  and  the  field  visit  in  September  2007. 

The  white  paper  on  point  classification  (Appendix  C)  was  based  primarily  on  data  from  the 
Former  Camp  Beale  demonstration  site,  along  with  discussion  with  vendors.  However,  its 
conclusions  were  verified  by  examining  lidar  data  from  the  other  ESTCP  demonstration  sites 
where  lidar  was  collected,  in  addition  to  lidar  data  from  two  non-ESTCP  sites. 

The  white  paper  on  lidar  error  was  based  primarily  on  published  literature  and  interviews,  and 
did  not  involve  site  selection. 

4.1.2  Activity  2:  Evaluate  Current  Software  Packages 

Data  from  portions  of  the  Kirtland  and  Victorville  demonstration  sites  were  used  for  the  software 
test.  The  Victorville  data  set  was  chosen  based  on  the  following  criteria: 

■  The  craters  at  this  target  are  large  (up  to  8  m  in  diameter),  deep  (up  to  1  m)  and  are 
generally  well  defined  in  the  lidar  data.  In  addition  to  overlapping  craters  in  patterns 
typical  of  crater  fields,  the  data  set  contains  many  isolated,  non-overlapping  craters. 

These  large,  isolated  craters  provided  a  good  “best  case”  data  set  for  an  initial  software 
test. 

■  Because  the  craters  are  large  and  well  defined,  a  comparison  data  set  against  which  to 
judge  the  software  results  can  be  created  with  a  high  degree  of  confidence  using  visual 
inspection  of  the  lidar  and  orthophotos. 
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The  Kirtland  data  set  included  Target  NDIA  and  Targets  N2  and  N3.  This  data  was  chosen 
based  on  the  following  criteria: 

■  Target  NDIA  is  a  high  explosive  bombing  target  containing  a  large  number  of  small  (2-3 
m  diameter),  shallow  (10-20  cm)  craters.  Because  these  craters  are  shallow  and  eroded, 
they  are  more  difficult  to  distinguish  from  the  normal  variation  in  the  ground  surface. 

This  data  constitutes  an  appropriate  complement  to  the  simpler  data  from  the  Victorville 
site. 

■  Targets  N2  and  N3  contain  large  targets  constructed  from  berms:  a  bull’s-eye  target  at 
N2,  and  a  bull’s-eye  target  and  a  ship  target  at  N3.  These  features  can  be  seen  clearly 
during  visual  inspection  of  the  lidar  data,  but  are  highly  eroded,  often  no  more  than  20  cm 
in  height,  and  are  not  visible  from  the  ground. 

4.1.3  Activity  3:  Summarize  Lessons  Learned  from  the  ESTCP  WAA  Pilot  Program 

The  deliverable  for  Activity  3  (Appendix  D)  summarizes  lessons  learned  at  the  applicable 
ESTCP  demonstration  sites. 

4.2  SITE  HISTORY 

The  history  and  characteristics  of  each  of  the  test  sites,  including  munitions  contamination,  are 
described  in  the  final  report  for  the  ESTCP  WAA  Pilot  Project  (Nelson  et  al.  2008).  Site  history 
is  not  available  for  the  non-ESTCP  sites  that  were  examined  in  the  white  paper  on  point 
classification  (Appendix  C). 

4.3  SITE  GEOLOGY 

Not  applicable. 
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5.0  TEST  DESIGN 

5.1  CONCEPTUAL  EXPERIMENTAL  DESIGN 

The  demonstration  was  executed  following  the  steps  shown  in  Figure  3. 


Figure  3:  Conceptual  Experimental  Design  Process 


Item 

Date  of  Completion 

Initial  planning  meetings 

6/15/2007 

Demonstration  plan,  initial 

7/1/2007 

Field  data  collection 

9/20/2007 

Analysis  of  vegetation  effects 

2/1/2008 

ESTCP  Symposium 

12/6/2007 

Demonstration  plan,  revised 

2/1/2008 

Feature  extraction  software,  downselect 

2/15/2008 

IPR 

2/28/2008 

Demonstration  plan,  revised  2 

4/1/2008 

White  paper  on  point  classification,  draft 

9/3/2008 

Guidance  document  preliminary  draft 

10/3/2008 

Interim  Progress  Report 

11/1/2008 

IPR 

10/7/2008 

ESTCP  Symposium 

12/4/2008 

White  paper  in  point  classification,  final 

2/1/2009 

White  paper  on  lidar  error,  draft 

5/1/2009 

Feature  extraction  software,  vendor  contact  and  responses 

5/1/2009 

IPR 

5/5/2009 

Guidance  document  revised  draft 

7/13/2009 

Final  Report 

1/31/2010 

5.1.1  Activity  1:  Systematically  Investigate  Vegetation  Effects 

Activity  1  was  designed  to  establish  performance  criteria  for  lidar  on  vegetated  sites  using 
existing  data.  Activity  1  consisted  of  four  steps. 

Step  1:  Classify  vegetation.  This  part  of  the  analysis  compared  the  density  and  height  of 
vegetation  using  lidar  data  to  density  and  height  measured  in  the  field  using  standard  forestry 
techniques.  The  field  crew  measured  vegetation  density  using  a  spherical  densitometer,  tree 
height  using  a  hand-held  clinometer,  and  grass  height  using  a  tape  measure  (see  Section  6.1.1). 
The  field  crew  examined  each  plot  to  determine  whether  there  were  ground  features  visible  that 
were  not  found  in  the  lidar  data,  or  features  in  the  lidar  data  that  were  not  found  in  the  field. 

Step  2:  Identify  ground  features.  This  part  of  the  analysis  examined  the  effects  of  lowering  the 
lidar  data  density  on  detection  of  ground  features.  The  lidar  data  was  decimated  (artificially 
thinned)  to  simulate  lower  data  collection  densities,  and  the  effects  of  this  decimation  on  feature 
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detection  were  examined.  URS  created  surface  models  at  selected  plots  using  the  full  lidar  data 
sets  and  four  decimated  data  sets,  using  the  orthophoto  data  for  the  site  as  a  cross-validation 
method  where  possible.  URS  then  examined  the  surface  models  visually  to  determine  whether 
the  ground  features  at  each  test  plot  was  visible  at  each  decimation  level. 

Step  3:  Produce  White  Paper  on  Lidar  Point  Classification  Methods.  At  the  In-Progress 
Review  meeting  in  February,  2008,  preliminary  results  were  presented  showing  that  in  some 
cases,  a  significant  number  of  lidar  points  on  paved  surfaces  had  been  classified  as  non-ground 
returns.  This  result  had  the  potential  to  lower  the  resolution  of  ground  surface  models.  ESTCP 
requested  a  technical  white  paper  examining  lidar  point  classification  methods,  determining 
whether  this  phenomenon  was  unique  to  a  single  data  set  or  vendor,  and  examining  potential 
changes  to  classification  methods.  The  white  paper  was  based  on  re-examining  data  from  all  of 
the  ESTCP  demonstration  sites  and  two  non-ESTCP  sites.  In  addition,  URS  conducted 
discussions  with  the  two  lidar  vendors  who  provided  data  for  the  ESTCP  demonstrations,  Terra 
Remote  Sensing  and  Sky  Research,  regarding  point  classification  methods  and  potential 
alternative  approaches.  Technical  peer  review  was  provided  by  Terra  Remote  Sensing,  Sky 
Research,  and  additional  vender  (Terrapoint,  LLC),  and  URS  internal  experts.  This  white  paper 
is  included  as  Appendix  C. 

Step  4:  Produce  White  Paper  on  Lidar  Error.  Following  the  October  2008  In-Progress  Review 
meeting,  ESTCP  requested  an  additional  technical  white  paper  summarizing  the  sources  and 
magnitudes  of  lidar  error  and  their  implications  for  the  use  of  lidar  at  military  munitions  sites. 
This  technical  paper  was  based  on  existing  literature;  no  original  investigation  was  conducted. 
The  white  paper  includes  consideration  of  both  lidar  points  and  digital  surface  models. 

Technical  peer  review  was  provided  by  Terra  Remote  Sensing,  Sky  Research,  and  additional 
vender  (Terrapoint,  LLC),  and  URS  internal  experts.  This  white  paper  is  included  as  Appendix 
B. 

5.1.2  Activity  2:  Evaluate  Current  Software  Packages 

Step  1:  Evaluate  existing  commercial  off-the-shelf  feature  extraction  software  packages.  URS 
examined  approximately  850  software  products  that  advertised  some  ability  to  manipulate  three- 
dimensional  spatial  data.2  Criteria  for  further  selection  included: 

■  Will  the  software  load  and  manipulate  lidar  point  formats  easily? 

■  Will  the  software  handle  large  lidar  data  sets?  What  are  the  file  size  limits? 

■  What  kind  of  processing  speeds  will  the  software  achieve  for  creating  surface  models? 

■  Will  the  software  output  results  be  compatible  with  common  GIS  and  CADD  software? 


2  The  source  of  the  original  list  of  software  products  was  compiled  by  the  US  Army  Topographic  Engineering 
Center,  which  compiled  “a  summary  of  3D  terrain  visualization  software  that  may  be  relevant  to  LiDAR/ASLM 
users.”  Originally  available  at:  http://www.tec.army.mil/TD/tvd/survev/index.html. 
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Based  on  this  survey,  URS  selected  17  software  packages  for  further  evaluation.  The 
manufacturers  of  the  selected  software  packages  were  contacted  and  invited  to  participate  in  a 
test  using  a  sample  lidar  data  set  containing  data  from  both  the  Kirtland  and  Victorville 
demonstration  sites.  Software  vendors  were  invited  to  report  the  number  and  type  of  features 
extracted,  and  to  describe  their  capabilities,  particularly  for  automated  extraction. 

Step  2:  Testing  software  packages.  Manufacturers  of  software  packages  that  successfully 
identified  features  during  Step  1  were  to  be  invited  to  participate  in  a  test  using  a  larger  and  more 
complex  lidar  data  set.  Based  on  the  results  of  Step  1,  URS  and  the  US  Army  Corps  of 
Engineers  recommended  that  this  step  not  be  undertaken  (see  Section  6.2). 

5.1.3  Activity  3:  Summarize  Lessons  Learned  from  the  ESTCP  WAA  Pilot  Program 

In  October  2008,  URS  submitted  a  preliminary  draft  guidance  document  to  ESTCP  summarizing 
the  lessons  learned  through  the  WAA  Pilot  Program  related  to  lidar  and  orthophotos,  as  well  as 
the  preliminary  results  from  the  first  two  activities  of  this  demonstration.  In  July  2009,  URS 
submitted  a  revised  draft  of  the  guidance  document  to  ESTCP  and  technical  peer  reviewers.  The 
final  version  appears  as  Appendix  D. 

5.2  SITE  PREPARATION 

Not  applicable. 

5.3  SYSTEM  SPECIFICATION 

Lidar  system  components  are  described  in  the  white  paper  on  lidar  error  (Appendix  B). 

5.4  CALIBRATION  ACTIVITIES 

Calibration  activities  are  described  in  the  final  reports  for  the  ESTCP  demonstration  sites  (URS 
2007  and  2008a)  and  the  ESTCP  Pilot  Project  Wide  Area  Assessment  for  Munitions  Response 
final  report  (Nelson  et  al.  2008). 

5.5  DATA  COLLECTION 

Lidar  data  collection  is  described  in  the  final  reports  for  each  of  the  demonstration  sites  (URS 
2007  and  2008a).  Terra  Remote  Sensing  collected  lidar  data  at  the  Former  Camp  Beale 
demonstration  site  between  September  10  and  13,  2007. 

For  field  studies,  URS  selected  51  test  plots  in  the  Spenceville  Wildlife  Refuge  in  the  southeast 
portion  of  the  Former  Camp  Beale  demonstration  site.  This  area  was  selected  because  it 
contained  a  full  variety  of  vegetation  coverage  ranging  from  bare  ground  with  no  grass  or  tree 
coverage  to  areas  with  field  grass  to  areas  with  near-complete  tree  canopy  closure.  The 
Spenceville  Wildlife  Refuge  is  also  public  land,  and  site  access  was  more  easily  obtained  than 
for  the  remainder  of  the  demonstration  site.  The  test  plots  were  circular  and  15  meters  in 
diameter. 
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The  site  visit  team  consisted  of  a  staff  member  from  the  US  Forest  Service  with  background  in 
evaluating  percentage  of  crown  cover  using  standard  forestry  techniques,  a  field  technician  to 
record  the  plot  centers  using  resource-grade  GPS  and  take  site  photos,  and  a  UXO  safety  officer. 

At  each  location,  the  site  team  recorded  the  date  and  time  of  the  visit;  percentages  of  brush,  grass 
and  tree  cover;  terrain  slope;  average  height  in  inches  of  any  grass  or  brush  present;  and  took 
notes  regarding  features  observed  on  the  ground.  The  site  team  also  took  digital  photos  at  each 
site  facing  north,  south,  east,  and  west.  During  a  later  site  visit  in  April  2009,  the  team  measured 
tree  heights,  which  were  not  measured  during  the  first  site  visit. 

5.6  VALIDATION 

The  final  reports  for  each  of  the  demonstration  sites  (URS  2007  and  2008a)  and  the  ESTCP  Pilot 
Project  Wide  Area  Assessment  for  Munitions  Response  final  report  (Nelson  et.  al.  2008) 
describe  validation  activities  for  the  lidar  data. 
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6.0  DATA  ANALYSIS  RESULTS 

6.1  ACTIVITY  1 :  SYSTEMATICALLY  INVESTIGATE  VEGETATION  EFFECTS 

6.1.1  Step  1:  Classify  Vegetation 
Vegetation  Density 

This  activity  compared  vegetation  density  measured  using  standard  field  methods  to  that 
measured  using  the  lidar  data.  The  objective  was  to  understand  whether  vegetation  density  as 
measured  by  lidar  would  correlate  with  vegetation  density  as  measured  using  standard  forestry 
instruments  and  methods.  URS  took  field  measurements  at  50  circular  15-m  diameter  test  plots 
using  a  spherical  densitometer,  an  instrument  that  uses  a  curved  mirror  divided  into  a  grid.  The 
technician  viewed  the  sky  overhead  in  the  mirror  and  counted  the  number  of  grid  cells  obscured 
by  vegetation.  The  result  is  given  as  a  percentage  of  crown  closure;  that  is,  the  percentage  of  sky 
obscured  by  foliage.  The  spherical  densitometer  was  chosen  as  a  standard,  low-cost  forestry 
instrument  most  likely  to  be  available  to  site  managers. 

For  lidar,  the  vegetation  density,  or  percentage  of  crown  closure,  was  derived  through  a  two-step 
process.  First,  lidar  points  representing  the  crown  were  classified  by  identifying  and 
reclassifying  all  points  within  0.4  m  of  the  modeled  ground  surface.  The  remaining  points 
represented  returns  from  higher  vegetation,  and  it  appeared  from  visual  inspection  that  the  vast 
majority  of  these  points  were  from  the  crown.  Figure  4  shows  an  example  of  the  lidar  points  at 
one  of  the  test  plots. 

Next,  the  percentage  of  crown  points  was  calculated  based  on  a  grid  of  3-m  cells.  This  cell  size 
was  selected  as  best  matching  the  field  conditions.  When  tested,  smaller  cells  yielded  too  many 
cells  with  very  low  values  due  to  gaps  in  the  foliage,  while  larger  cells  were  too  large  to  show 
any  variation  at  all.  For  each  cell,  the  number  of  crown  and  non-crown  points  was  counted,  and 
percentages  computed.  Figure  5  shows  an  example  of  a  vegetation  density  map  for  one  of  the 
test  plots. 

Finally,  for  each  plot,  an  average  vegetation  density  was  derived,  based  on  the  percentage  of 
crown  points  for  the  cells  in  the  plot.  These  values  were  compared  to  the  crown  closure  values 
measured  in  the  field.  Results  are  shown  in  Figure  6. 

Figure  6  shows  that  correlation  between  lidar  and  field  methods  was  reasonably  good  at  lower 
data  densities  but  that  as  vegetation  density  increased,  lidar  recorded  lower  vegetation  densities 
than  field  measures.  This  result  may  be  because  of  the  difference  in  the  visual  perception  of 
crown  closure  and  the  ability  of  lidar  to  penetrate  small  openings  in  the  vegetation.  This  finding 
is  consistent  with  the  common  observation  that  lidar  will  penetrate  what  at  first  appears  to  be 
very  dense  vegetation.  In  fact,  such  vegetation  usually  contains  numerous  small  gaps  where 
laser  signals  can  penetrate  to  the  ground. 
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Figure  4:  Example  Lidar  Point  Cloud 


In  the  image  to  the  left,  the  lidar  points  are  color  coded  as: 
Blue:  vendor-classified  ground  points 
Orange:  points  within  0.4  m  of  the  vendor’s  ground  surface 
Green:  remaining  points  assumed  to  be  higher  vegetation 
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Figure  5:  Example  Vegetation  Density  Map 
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Figure  6:  Vegetation  Density,  Lidar  vs.  Field  Methods 


Tree  Heights 

During  this  part  of  the  study,  URS  compared  tree  heights  based  on  field  measurements  to  those 
based  on  lidar,  in  order  to  confirm  that  site  managers  could  use  lidar  to  obtain  relatively  accurate 
tree  height  measurements.  URS  performed  field  measurement  at  the  50  test  plots  using  a 
common  trigonometric  method  used  by  foresters  (Figure  7).  Angles  to  the  top  of  the  tree  and 
base  of  the  tree  were  measured  using  a  clinometer,  a  device  that  measures  the  angle  above  or 
below  horizontal  from  a  measured  distance  away.  The  combination  of  the  angle  and  the  distance 
from  the  tree  yields  the  height  of  the  tree  through  simple  trigonometry.  Distances  were  measured 
using  a  tape  measure.3 


3  More  accurate  methods  for  measuring  tree  heights  are  available;  for  example,  using  laser  range  finders.  However, 
the  objective  of  this  study  was  to  mimic  low-cost  tools  and  methods  most  likely  to  be  available  to  site  managers.  A 
positive  result  with  these  methods  would  eliminate  the  need  for  more  costly,  if  more  accurate,  methods. 
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Field  heights  were  compared  to  two  different  lidar  height  measurements.  For  the  first,  URS 
created  a  digital  surface  model  (DSM)  based  on  all  lidar  points,  using  aim  cell  size.  Cell 
values  were  interpolated  from  the  heights  of  the  lidar  points  above  the  ground  surface  (not  their 
absolute  elevations),  and  the  cell  with  the  highest  value  in  the  plot  was  taken  to  represent  the  tree 
height.  For  the  second  measurement,  the  lidar  point  with  the  highest  height  above  the  ground 
surface  in  the  plot  was  taken  as  the  tree  height. 

Figure  8  shows  that  the  DSM  provided  a  very  poor  correlation  with  field  heights,  but  using  the 
highest  lidar  point  method  showed  good  correlation.  This  result  is  consistent  with  numerous 
studies,  which  have  shown  that  lidar  can  provide  accurate  measurements  of  individual  tree 
heights  in  both  deciduous  and  coniferous  forests  (Andersen  et  al.  2006,  Lim  et  al.  2001). 

Grass  Heights 

During  this  part  of  the  study,  URS  compared  the  height  of  the  tall  grass  under  the  trees  based  on 
field  measurements  to  those  based  on  lidar.  Field  teams  used  a  tape  measure  to  determine  the 
average  grass  height  in  the  field.  Lidar  heights  were  estimated  using  the  lidar  points  less  than  0.4 
m  from  the  modeled  ground  surface.  Inspecting  the  point  clouds  showed  that  these  low  points 
are  likely  to  include  the  vast  majority  of  the  returns  from  the  grass  area,  and  that  lidar  points 
above  this  height  are  most  likely  to  be  from  tree  trunks  or  branches  (Figure  9). 

Grass  heights  were  estimated  by  using  the  highest  “low  vegetation”  values  in  the  plot.  Figure  10 
shows  that  lidar  consistently  and  substantially  under-measured  the  height  of  the  grass  compared 
to  field  measurement.  It  is  possible  that  the  relatively  diffuse  grass  on  the  Former  Camp  Beale 
site  did  not  have  sufficient  mass  to  cause  a  laser  reflection  until  closer  to  the  ground. 
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Figure  8:  Tree  Heights,  Lidar  vs.  Field  Methods 
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Tree  heights,  lidar  compared  to  field  methods.  Left:  results  from  lidar  surface  model  method,  right:  results  from  highest  lidar  point 

method. 


Figure  9:  Example  Grass  Conditions  and  Lidar  Point  Cloud 


Example  test  plot,  orthophoto  and  lidar  point  cloud. 


Page  28  of  71 


Project  Number  07  E-MM2-012/  MM-0737 
Development  of  Parameters  for  the  Collection  and 
Analysis  of  Lidar  at  Military  Munitions  Sites 
Draft  Final  Report 


Figure  10:  Grass  Heights,  Lidar  vs.  Field  Methods 
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Only  a  few  published  studies  were  found  on  using  lidar  to  measure  low  vegetation  heights. 
Ritchie  (1996)  reported  good  agreement  between  lidar  measurement  and  field  measurement  for 
desert  scrub.  However,  the  quoted  studies  were  broad-level  hydrologic  studies  and  may  not  be 
comparable  to  the  Former  Camp  Beale  site.  Gene  et  al.  (2004)  reported  good  agreement  between 
lidar  and  field  measurement  of  wetland  vegetation  in  Florida,  including  low  vegetation. 

However,  the  vegetation  in  question  was  not  identified  by  species  and  did  not  appear  to  include 
grass. 

6.1.2  Step  2:  Identify  Ground  Features 
Data  Collection 

During  this  part  of  the  study,  URS  re-examined  the  Former  Camp  Beale  data  and  examined  the 
relationship  between  vegetation  density  and  the  size  of  features  detected.  Work  at  the  Former 
Camp  Beale  site,  as  well  as  experience  in  the  lidar  industry  generally,  has  shown  that  lidar  can 
reveal  features  under  tree  cover.  At  the  Former  Camp  Beale  site,  a  variety  of  ground  features 
were  detected  under  trees,  with  the  smallest  features  visible  about  1  m  in  diameter.  Figure  1 1 
shows  some  representative  features. 
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Figure  11:  Example  Feature  Detection  under  Trees 


1.5  m  feature,  orthophoto  and  lidar  surface  model 


3.3  m  feature,  orthophoto  and  lidar  surface  model 
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3.9  m  feature,  orthophoto  and  lidar  surface  model 


8.0  m  (in  width)  feature,  orthophoto  and  lidar  surface  model 


During  the  field  visit,  the  team  visited  59  test  plots,  and  took  digital  photos  of  52  features  in 
vegetation  conditions  ranging  from  completely  open  to  the  most  dense  tree  cover  on  the  site. 
Seven  test  plots  had  no  visible  features.  No  additional  features  were  discovered  that  were  not 
visible  in  the  lidar  data,  indicating  that  lidar  at  the  full  data  density  had  detected  all  of  the  ground 
features  in  the  plots  that  could  be  seen  by  observers  on  the  ground. 
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Effects  of  Lowering  Point  Density  on  Feature  Detection 

During  this  part  of  the  study,  URS  examined  the  effect  of  lowering  the  lidar  point  density  on 
feature  detection.  The  objective  was  to  determine  a  lidar  point  density  below  which  ground 
features  would  cease  to  be  visible.  Fifty-two  features  were  examined  using  successively  lower 
lidar  point  density.  The  density  of  the  original  lidar  data  set  was  artificially  lowered  (decimated), 
using  an  algorithm  that  attributed  every  point  so  that  datasets  could  be  extracted  by  selecting 
every  second,  fourth  eighth  and  sixteenth  point.  Using  these  classifications,  lidar  datasets  were 
created  with  1/2,  1/4,  1/8,  and  1/16  of  the  original  data  density.  Choosing  points  in  this  way 
mimicked  the  action  of  a  slower  mirror  speed  and  pulse  rate.  New  ground  surface  models  were 
created  using  each  of  the  four  new  datasets.4 

Once  the  points  were  classified  by  decimation  level,  URS  created  new  surface  models  and 
evaluated  the  visibility  of  the  features  in  each.  Three  levels  of  visibility  were  used:  clearly 
visible  (green),  barely  visible  (yellow),  and  not  visible  (red).  These  categories  reflect  the 
somewhat  subjective  nature  of  the  determination.  Table  3  shows  a  sample  of  the  output  of  this 
exercise,  and  the  results  for  all  52  test  plots  are  included  in  Appendix  E. 

Table  3:  Sample  Analysis  Output 


m  -  meter 

pts/m2  -  points  per  square  meter 


As  expected,  feature  detection  decreased  with  decreasing  lidar  point  density.  However,  this 
decrease  was  uneven  and  varied  considerably  from  one  feature  to  the  next.  At  one  extreme,  the 
visibility  of  some  features  declined  evenly  with  decrease  in  data  density  (Figures  12  and  13).  At 
the  other  extreme,  some  features  barely  degraded  at  all  as  density  was  lowered  (Figures  14  and 
16),  even  though  the  surrounding  surface  degraded  significantly.  This  result  is  logical  since  even 
very  low-density  lidar  data  will  still  reveal  features  if  the  lidar  points  happen  to  fall  in  and  near 
the  feature.  Images  of  each  plot  showing  the  orthophoto  and  the  five  surface  models  are 
included  in  Appendix  F,  Feature  Detection  and  Point  Decimation  -  Surface  Model  Results. 


4  For  this  demonstration,  the  original  classifications  as  ground  or  non-ground  returns  from  the  full  data  set  were 
retained. 
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Figure  12:  Features  Showing  Steady  Degradation,  Example  1 
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Figure  13:  Features  Showing  Steady  Degradation,  Example  2 


Test  plots,  orthophotos 


Level  1,  original  data.  Average  lidar  data  density:  13.8 
pts/m2 


Level  3,  average  lidar  data  density:  3.4  pts/m2 


Level  4,  average  lidar  data  density:  1.7  pts/m2 


Level  2,  average  lidar  data  density:  6.9  pts/m2 


Level  5,  average  lidar  data  density:  0.8  pts/m2 
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Figure  14:  Features  Showing  Little  or  No  Degradation,  Example  1 


Test  plots,  orthophotos 


Level  1,  original  data.  Average  lidar  data  density:  13.8 
pts/m2 


Level  2,  average  lidar  data  density:  6.9  pts/m2 
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Figure  15:  Features  Showing  Little  or  No  Degradation,  Example  2 


Test  plots,  orthophotos 


Level  3,  average  lidar  data  density:  3.4  pts/m2 


Level  1,  original  data.  Average  lidar  data  density:  13.8 
pts/m2 


Level  4,  average  lidar  data  density:  1.7  pts/m2 


Level  2,  average  lidar  data  density:  6.9  pts/m2 


Level  5,  average  lidar  data  density:  0.8  pts/m2 
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Figure  16  summarizes  the  results  for  the  52  features  examined.  Significant  loss  of  feature 
detection  begins  to  appear  between  level  2  (6.9  points  per  square  meter  [pts/m2])  and  level  3  (3.4 
pts/m2),  with  over  two-thirds  of  the  features  not  visible  by  level  5  (0.8  pts/m2).  There  was  some 
correlation  between  the  rate  of  degradation  and  the  size  of  the  feature;  that  is,  larger  features 
tended  to  disappear  more  slowly.  However,  this  correlation  was  not  uniform. 


Figure  16:  Data  Density  and  Feature  Detection  Summary 


■  Not  Visible 

□  Partial 

□  Visible 


Density  Level 


6.1.3  Step  3:  Produce  White  Paper  on  Lidar  Point  Classification  Methods 

The  final  version  of  this  white  paper  appears  as  Appendix  C.  Principal  findings  are  summarized 
here. 

In  all  lidar  investigations,  an  important  task  is  to  distinguish  laser  signals  that  return  from  the 
ground  from  those  that  return  from  vegetation,  buildings,  or  other  objects.  This  task  is  carried 
out  through  the  use  of  automated  algorithms  followed  by  inspection  and  editing  by  skilled 
operators. 

This  problem  is  most  difficult  in  the  case  of  lidar  points  close  to  the  ground  surface.  Identifying 
returns  from  trees  is  relatively  straightforward,  but  low  brush  and  grass  are  more  difficult,  and 
results  depend  more  on  operator  experience  and  judgment.  For  the  lowest  elevation  lidar  points, 
it  can  become  very  difficult  to  distinguish  laser  returns  from  an  uneven  ground  surface  or  from 
low  brush  or  grass.  The  problem  of  point  classification  on  uneven  ground  surfaces  has  been 
observed  for  many  years,  as  illustrated  in  Figure  17.  Frequently,  laser  returns  are  classified  as 
non-ground  when  orthophotos  and  other  evidence  indicate  that  they  are  most  likely  ground 
returns  from  uneven  surfaces.  In  some  cases,  points  will  be  classified  as  non-ground  even  when 
they  are  returned  from  relatively  flat,  un-vegetated  surfaces.  At  the  ESTCP  demonstration  sites, 
the  phenomenon  of  laser  returns  incorrectly  classified  as  non-ground  points  appeared,  for 
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instance,  in  the  incidence  of  points  classified  as  non-ground  that  were  returned  from  road 
surfaces  (Figure  18). 

Figure  17:  Lidar  Point  Classification  on  Forest  Roads 


Lidar  data  collected  in  2001  in  a  forest  and  timber  harvest  area  in  southeast  Alaska.  Green  points, 
classified  as  non-ground  returns,  can  be  seen  on  the  surface  of  the  forest  road  where  the  photo 
shows  no  vegetation.  It  is  difficult  to  determine  whether  these  “non-ground”  returns  are  artifacts 
of  the  classification  process  or  result  from  the  uneven  surface  of  the  dirt  road. 


Figure  18:  Lidar  Classification  Results  on  Paved  Road  Surfaces 


Orthophoto  and  classified  lidar  points  for  area  portion  of  the  Former  Camp  Beale  demonstration  site.  At  right,  the  vendor’s 


Page  38  of  71 


Project  Number  07  E-MM2-012/  MM-0737 
Development  of  Parameters  for  the  Collection  and 
Analysis  of  Lidar  at  Military  Munitions  Sites 
Draft  Final  Report 


ground  points  are  black,  and  the  vendor’s  non-ground  points  are  green.  The  road,  which  should  not  have  any  non-ground 
returns,  shows  many  points  classified  as  non-ground. 


In  response  to  this  observation,  ESTCP  requested  production  of  a  white  paper  (Appendix  C) 
describing  the  point  classification  process,  with  the  goal  of  understanding  whether  this 
phenomenon  was  peculiar  to  the  one  data  set  examined  or  to  the  work  of  a  single  vendor,  or  was 
common  to  lidar  data  processing  generally. 

URS  found  that  the  phenomenon  of  over-classifying  non-ground  points  was  common  to  all  of  the 
ESTCP  lidar  data  sets  and  both  of  the  vendors  used,  and  to  the  two  non-ESTCP  lidar  data  sets 
collected  by  other  vendors.  The  percentage  of  non-ground  classifications  on  road  surfaces  varied 
widely,  ranging  from  12%  to  over  95%.  The  percentage  of  lidar  points  misclassified  as  non¬ 
ground  was  higher  in  flight  line  overlap  areas  and  other  areas  where  the  local  density  of  lidar 
points  was  higher.  As  the  overall  density  of  lidar  points  increased,  the  rate  of  misclassification 
also  increased.  This  higher  rate  of  misclassification  eliminated  some  of  the  advantage  of 
collecting  lidar  at  a  higher  density,  since  it  was  specifically  the  higher-density  areas  where 
misclassification  was  most  common. 

Misclassification  of  ground  points  was  found  to  result  from  the  interaction  of  standard  software 
settings  with  the  vertical  error  present  in  the  lidar  data.  Classification  methods  rely  in  part  on  the 
angles  between  adjacent  lidar  points,  with  higher  angles  being  used  as  an  indicator  of  a  non¬ 
ground  reflection.  In  areas  of  high  point  density  the  lidar  points  are  closer  together,  and  even 
small  height  differences  between  points  can  be  sufficient  to  cause  the  higher  point  to  be 
classified  as  non-ground.  Where  lidar  points  are  particularly  close  together,  even  height 
differences  well  within  the  15-cm  vertical  accuracy  specification  can  exceed  the  classification 
parameters  (Figure  19). 

Figure  19:  Example  Iteration  Angles 


1  meter  (100  cm)  point  spacing 


20  cm  point  spacing 
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This  phenomenon  is  described  in  greater  detail  in  Appendix  C. 

The  over-classification  of  non-ground  returns  does  not,  in  itself,  address  the  issue  of  the  source 
of  the  vertical  offsets  which  with  the  classification  software  is  interacting.  One  vendor  contacted 
pointed  out  that  misclassification  would  be  a  predictable  effect  of  errors  in  GPS  or  flight  line-to- 
flight  line  data  calibration,  either  of  which  would  result  in  larger  than  necessary  vertical 
discrepancies  between  points.  This  vendor  noted  that  if  the  flight  lines  are  vertically  offset  (due 
to  calibration  or  GPS  errors),  then  points  associated  with  the  higher  flight  line  would  be 
predictably  classified  as  non-ground.  This  effect  would  be  present  even  if  the  classification 
routine  were  well  constructed.  Reducing  the  misclassification  effect  should  therefore  include 
checks  for  successful  calibration  and  GPS  quality.  (Sky  Research,  2009).  Other  vendors 
concurred. 

However,  better  calibration  will  not  eliminate  the  residual  inherent  error  between  individual  lidar 
points  or  lines  of  lidar  points,  which  may  lead  to  some  misclassification  depending  on  the 
parameters  set  up  in  the  classification  routine  and  the  distance  between  the  laser  returns. 
Consequently,  careful  examination  of  point  classification  methods  will  remain  desirable 
whenever  lidar  is  used  to  detect  and  discriminate  small  ground  features. 

Adjusting  the  point  classification  parameters  is  easily  accomplished  by  all  vendors,  and 
classification  can  be  adjusted  based  on  the  needs  of  each  particular  lidar  acquisition. 
Alternatively,  the  vendor  can  use  their  standard  procedures,  but  also  establish  an  additional 
classification  for  low  lidar  points.  These  points  can  be  added  to  the  ground  surface  model  at  a 
later  date  in  order  to  increase  the  resolution  of  the  surface  model.  A  similar  reclassification  can 
be  accomplished  after  the  fact  using  standard  GIS  methods. 

URS  tested  the  effects  of  re-classifying  all  lidar  points  within  40  cm  of  the  modeled  ground 
surface  at  59  test  plots  at  the  Former  Camp  Beale  demonstration  site.  Adding  the  re-classified 
points  back  into  the  ground  surface  model  increased  the  clarity  of  the  ground  features,  and 
allowed  the  detection  of  additional  small  features  at  36%  (21)  of  59  test  plots  (Figures  20  and 
21). 
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Figure  20:  Sample  Point  Reclassification  Results 


Initial  point  classification  test  area,  points  re-classified.  Non-ground  points  within  0.4  m  of  the  modeled  ground  surface  were 
added  to  the  ground  classification.  The  remaining  non-ground  points  appear  to  be  reflections  from  trees  and  other  objects  on 
the  ground.  The  image  to  the  right  is  the  digital  surface  model  using  the  full  lidar  data  set. 

Adding  all  of  the  low  points  into  the  surface  model,  as  was  done  for  this  test,  also  will  add  small 
surface  features  such  as  fallen  tree  trunks  on  the  ground  surface,  which  are  not  truly  ground 
features.  This  was  clearly  evident  at  the  Former  Camp  Beale  demonstration  site,  especially  when 
smaller  grid  cells  were  used  (Figure  22).  This  method  also  will  also  add  instrument  noise,  in  the 
form  of  surface  roughness  resulting  from  vertical  error  in  the  lidar  points  that  would  be  removed 
by  standard  approaches.  However,  noise  did  not  noticeably  change  the  appearance  of  the  surface 
models  at  the  test  plots. 

The  objective  of  UXO/MEC  surveys  is  to  detect  objects  such  as  small  craters.  The  major  finding 
of  the  white  paper  was  that  re-classifying  the  lidar  points  had  potential  to  increase  the  resolution 
of  the  ground  surface  model  and  the  detection  of  small  ground  features.  These  results  underscore 
the  importance  of  adjusting  the  classification  methods  used  for  each  site,  experimenting  with  re¬ 
classification  after  the  lidar  data  is  delivered,  and  discussing  point  classification  methods  with 
the  vendor  in  order  to  maximize  the  possibility  of  detection. 
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Figure  21:  Surface  Models  Using  Reclassified  Points 


Plot  3:  Vendor-classified  points,  0.3m  grid  size.  Tree  trunks 
are  faintly  visible. 


Plot  3:  Reclassified  points,  0.3m  grid  size.  Many  more  tree 
trunks  are  visible,  and  all  are  more  clearly  defined. 
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6.1.4  Step  4:  Produce  White  Paper  on  Lidar  Error 

The  final  version  of  this  white  paper  appears  as  Appendix  B.  Principal  findings  are  summarized 
here. 

Error  in  Lidar  Points 

The  positional  error  of  lidar  points  is  divided  into  horizontal  (x,  y)  and  vertical  (z)  error,  and  is 
typically  expressed  as  the  difference  between  lidar  values  and  surveyed  points  established  in  the 
study  area.  A  typical  vertical  accuracy  specification  is  5-10  cm  for  hard  surfaces  and  open 
regular  terrain;  15  cm  for  soft  or  vegetated  surfaces  and  flat  to  rolling  terrain;  and  from  30-50 
cm  on  more  extreme  terrain  (Ambercore  2008).  Horizontal  accuracy  is  typically  50-75  cm  in  all 
but  extremely  steep  terrain. 

Positional  error  in  lidar  points  results  primarily  from  a  combination  of  equipment  error  and  error 
resulting  from  site  conditions.  For  equipment  error,  manufacturers  and  vendors  typically  report 
the  positional  error  of  lidar  points  for  the  entire  lidar  system,  rather  than  for  individual  system 
components.  When  the  system  is  calibrated  and  functioning  correctly,  equipment  error  varies 
with  two  main  factors:  the  height  of  the  sensor  above  the  ground  and  the  angle  of  the  laser  signal 
off  directly  vertical  (nadir).  A  typical  lidar  system  error  curve  is  shown  in  Figure  23  for  the 
Leica  ALS60,  a  lidar  system  in  common  use.  Figure  23  shows  that  horizontal  (x,y)  error  is 
larger  than  vertical  error,  and  increases  more  sharply  than  vertical  error  (z)  with  increasing 
height.  Most  lidar  surveys  at  munitions  sites  are  flown  near  at  or  below  1 ,000  m,  where 
equipment  error  is  relatively  low. 

Figure  23:  Leica  ALS60  Horizontal  and  Vertical  Error  Curve 
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Flight  Height  (m) 


Source:  see  Appendix  B. 
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A  wide  variety  of  site  conditions  can  contribute  to  positional  error  in  lidar  data.  Some  conditions 
impact  the  entire  data  set,  either  by  affecting  the  equipment  directly  or  interfering  with  the  GPS 
signal.  Others,  such  as  terrain  roughness,  may  impact  data  from  only  part  of  the  surveyed  area. 
Site  conditions  that  contribute  the  largest  error  to  lidar  positions  include: 

Slope.  Lidar  collected  on  steep  terrain  will  be  less  accurate  than  that  on  flat  terrain  (Figure  24). 

Figure  24:  Effects  of  Slope  on  Lidar  Point  Error 


Potential 
Horizontal  Error 


Slope  compounds  both  horizontal  and  vertical  error  of  lidar  points. 


This  error  arises  in  two  ways: 

■  The  lidar  pulse  spreads  as  the  beam  travels.  The  degree  of  beam  divergence  is  affected 
by  the  characteristics  of  the  laser  combined  with  the  flight  height,  with  typical  beam 
footprints  between  10  and  100  cm  in  diameter.5  Beam  divergence  creates  an  area  of 


5  Formula  for  beam  divergence  is  roughly:  spot  diameter  at  nadir  =  elevation  (meters)  *beam  divergence  (radians) 
(Baltsavias  1999).  System  manufactures  report  beam  divergence  factors  from  0.22  to  1.0  mrad  (Key  2009).  Laser 
footprints  would  therefore  range  from  22  to  100  cm  at  1,000  m  flight  elevations. 
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uncertainty  as  to  the  exact  point  where  enough  energy  is  reflected  to  trigger  a  return.  On 
flat  surfaces,  this  uncertainty  may  affect  the  horizontal  location  of  the  return,  but  will  not 
affect  the  elevation  value.  On  sloping  surfaces,  the  area  of  the  pulse  footprint  will  be 
larger,  and  also  will  include  a  vertical  error  range. 

■  The  horizontal  error  of  the  lidar  location  will  be  larger  on  sloped  ground,  and  this 
horizontal  error  will  magnify  the  vertical  error  compared  to  flat  ground. 

Ground  surface.  Some  vendors  guarantee  higher  accuracy  on  hard  surfaces  than  on  soft  ground 
surfaces  based  on  the  fact  that  soft  surfaces  are  often  less  clearly  definable.  Plowed  fields  or 
grassy  areas,  for  instance,  can  have  furrows  or  vegetation  that  would  create  error  up  to  10-20  cm 
(Terra  Remote  Sensing  2009). 

Similarly,  occasionally  highly  reflective  surfaces  will  appear  to  be  slightly  raised  in  comparison 
to  non-reflective  surfaces,  such  as  white  painted  centerlines  on  asphalt  roadways.  In  such  cases, 
it  is  likely  that  more  reflective  surfaces  return  sufficient  energy  to  trigger  a  response  more 
quickly  than  less  reflective  surfaces. 

Vegetated  conditions.  Under  vegetated  conditions,  it  is  common  for  the  majority  of  laser  signals 
to  reflect  from  the  vegetation  rather  than  the  ground  surface.  The  lower  density  of  lidar  points 
under  vegetation  can  lead  to  apparently  lower  accuracies  of  the  individual  lidar  points.  This  is 
not  because  the  accuracy  of  the  individual  lidar  points  is  lower.  Rather,  the  lower  density  of 
ground  returns  in  vegetated  areas  means  that  survey  points  in  such  conditions  will  be  compared 
to  a  coarser  surface  model. 

Electromagnetic  interference.  Certain  types  of  electromagnetic  signals  can  affect  lidar  data 
collection,  usually  by  interfering  with  the  GPS  signal.  This  can  include: 

■  The  on-board  radio  system 

■  High-intensity  radar  in  the  immediate  area 

■  Geomagnetic  activity,  usually  as  a  result  of  solar  flares  and  other  solar  activity 
Surface  Model  Creation  and  Display  Methods 

In  practice,  much  of  lidar  data  analysis  is  performed  using  surface  models  derived  from  the  lidar 
points.  The  typical  final  products  of  lidar  are  digital  models  of  the  ground  surface,  both  of  the 
bare  ground  and  the  ground  with  vegetation  and  buildings  included.  Typical  model  types  are 
digital  elevation  models  (DEMs),  DSMs,  and  digital  terrain  models  (DTMs). 

A  DEM,  as  defined  by  the  US  Geological  Survey,  is  a  digital  file  consisting  of  terrain  elevations 
for  ground  positions  at  regularly-spaced  horizontal  intervals  that  portrays  the  ground  surface  free 
of  vegetation  or  human-created  structures.6  DEMs  may  be  created  using  data  from  many  sources 
in  addition  to  lidar,  including  topographic  maps,  ground  survey,  photogrammetry,  or  synthetic 
aperture  radar.  In  the  context  of  lidar,  the  DEM  is  the  product  through  which  the  irregularly- 


6  See:  http://rockyweb.cr.usgs.gov/elevation/dpi_dem.html 
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spaced  lidar  points  that  the  vendor  classifies  as  returning  from  the  ground  surface  are  converted 
to  a  regularly-spaced  grid  of  elevation  values.  This  digital  model  can  then  be  used  in  standard 
GIS  or  other  software  to  produce  hillshaded  surfaces,  contour  lines,  or  other  digital  products. 

Regularly  spaced  elevation  files  of  this  type  also  may  be  created  using  all  returns,  including 
those  from  buildings,  trees  and  other  features.  The  US  Geological  Survey  refers  to  these  as 
DSMs.  The  term  DTM  is  used  as  a  synonym  for  both  DEM  and  DSM.  In  the  lidar  context,  the 
term  DTM  is  most  frequently  used  to  refer  to  the  all-points  surface  model.  Appendix  B  uses  the 
term  DSM  for  the  all-points  surface  model  including  both  ground  and  non-ground  returns. 

Factors  Affecting  Surface  Models.  The  accuracy  and  usefulness  of  a  surface  model  will  depend 
on  the  accuracy  of  the  lidar  data  from  which  it  is  created,  and  will  be  subject  to  the  types  of  error 
that  affect  lidar  points.  However,  the  methods  used  to  create  the  the  model  can  impact  both  its 
accuracy  and  usefulness,  with  the  factors  examined  including  the  point  classification  methods, 
interpolation  method,  choice  of  cell  size  and  the  methods  for  displaying  hillshades. 

Point  classification  methods.  The  point  classification  approach  used  by  the  lidar  vendor  should 
have  no  effect  on  the  accuracy  or  precision  of  the  individual  lidar  points,  since  classification 
takes  place  after  the  data  is  calibrated.  However,  methods  that  are  biased  toward  creating  clean, 
smooth  ground  surfaces  can  result  in  a  lower  data  density  of  ground  points,  and  this  can  result  in 
lower  accuracy  compared  to  surveyed  control. 

Interpolation  method.  The  technical  problem  in  creating  surface  models  from  lidar  points  is  to 
assign  an  elevation  value  to  each  regular  grid  cell  from  the  semi-random  distribution  of  lidar 
points.  Various  interpolation  methods  exist,  and  the  method  should  be  adapted  to  the  specifics  of 
each  site.  Limited  testing  of  different  interpolation  methods  for  the  ESTCP  data  showed  only 
small  differences.  Further  discussion  of  interpolation  methods  is  found  in  Appendix  C  and  in 
High  Density  Lidar  and  Orthophotography  in  UXO  Wide  Area  Assessment  (URS  2007), 
especially  Appendix  D  of  that  document,  GIS-Based  Methods  for  Creating  Ground  Surface 
Models  from  Lidar  Points. 

Cell  size.  Choice  of  cell  size  can  affect  the  detection  of  small  surface  features  dramatically 
(Figure  25),  with  smaller  cell  sizes  performing  better. 
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Figure  25:  Effects  of  Digital  Surface  Model  Cell  Size  on  Feature  Detection 


Former  Camp  Beale  ESTCP  demonstration  site.  Hillshade 
and  DEM  based  on  0.3  m  cells.  Craters  are  approximately 
3  m  in  diameter. 


Hillshade  and  DEM  based  on  2.0  m  cells 


Hillshade  and  DEM  based  on  1.0  m  cells 


Hillshade  and  DEM  based  on  3.0  m  cells 


In  discussing  cell  size,  Smith  et  al.  (2004)  state  that  there  has  been  little  research  into  the  effect 
of  changing  grid  cell  size  other  that  that  by  Behan  (2000),  who  found  that  the  most  accurate 
surfaces  were  created  using  grids  that  had  a  similar  spacing  to  the  original  points.  This 
conclusion  makes  sense  intuitively  since  this  procedure  would  result  in  a  grid  where  each  cell 
had,  on  average,  at  least  one  measured  elevation.  Discussion  with  vendors  suggests  that  use  of 
cells  smaller  than  the  average  point  spacing  does  not  sacrifice  accuracy  (Kusevic  2009). 
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Using  this  logic,  a  ground  surface  model  based  on  a  lidar  data  set  with  approximately  two  ground 
points  per  meter  (the  average  density  of  ground  points  classified  by  the  vendor  for  the  Former 
Camp  Beale  site)  should  contain  approximately  two  cells  per  square  meter,  or  be  approximately 
0.7  m  on  a  side.  The  grid  cells  produced  by  URS  for  the  Kirtland,  Victorville,  and  Former  Camp 
Beale  sites  were  1  m,  and  those  produced  by  Sky  Research  for  the  Pueblo  site  were  0.5  m. 

Adding  more  reclassified  points  to  the  ground  surface  model  at  three  test  plots  at  the  Former 
Camp  Beale  site  roughly  doubled  the  number  of  ground  points  to  approximately  four  ground 
points  per  square  meter  (Table  4).  This  opens  the  possibility  of  reducing  the  size  of  the  grid 
cells.  At  this  density,  cell  size  equivalent  to  ground  point  spacing  would  be  0.5  m. 


Table  4:  Data  Densities,  Reclassified  Points 


Plot 

1 

2 

3 

Plot  reference  # 

3977 

4004 

4088 

Total  lidar  points 

3,761,158 

3,632,790 

3,832,829 

Vendor  ground  points 

456,659 

455,130 

470,594 

Reclassified  ground  points 

975,812 

898,548 

1,034,459 

All  points  density  (pts/m2) 

15.04 

14.53 

15.33 

Vendor  ground  points  density  (pts/m2) 

1.83 

1.82 

1.88 

Reclassified  ground  points  density 
(pts/m2) 

3.90 

3.59 

4.14 

As  a  test,  new  surface  models  were  created  for  these  three  plots  using  0.3  m  cells.  These  smaller 
cell  sizes  were  intended  as  a  preliminary  test,  and  were  probably  somewhat  smaller  than  justified 
by  the  densities  of  the  reclassified  points  (Figure  26). 

In  each  of  these  cases,  the  use  of  smaller  grid  cells  revealed  additional  features,  and  showed  the 
existing  features  with  greater  clarity.  This  result  amplifies  the  potential  benefit  of 
reclassification.  An  examination  of  surface  model  cell  size  should  be  included  in  the  analysis 
methods  considered  for  each  site. 
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Figure  26:  Effects  of  Changing  Grid  Cell  Size 


Reclassified  1  m  cells  Reclassified  0.3  m  cells 

The  smaller  grid  cell  size  shows  better  definition  of  small  features  and  some  possible  additional  features  in  the  center  right 

along  the  stream  course. 
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Test  plot  2,  orthophoto  and  original  ground  surface  model,  1  m  cells 


Reclassified  1  m  cells  Reclassified  0.3  m  cells 

The  smaller  grid  cell  size  shows  some  additional  features  in  the  upper  left  and  shows  the  existing  features  with  additional 

clarity. 


Surface  model  display  methods.  DEMs  and  DTMs  are  usually  displayed  as  hillshade  images  to 
enhance  the  visualization  of  surface  features  (Figure  27).  ArcGIS,  the  most  commonly  used  GIS 
software,  contains  dozens  of  settings  that  can  affect  the  usefulness  of  these  images,  as  do  other 
software  products  that  can  produce  hillshades.  Different  hillshade  settings  may  be  appropriate  in 
some  circumstances,  and  users  need  to  work  with  analysts  both  to  fully  understand  the  goals  of 
the  survey  and  to  experiment  with  appropriate  settings. 
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Figure  27:  Hillshade  Display  Method  Examples 


M»t*rs  ^  Mvtirs 


Former  Camp  Beale  ESTCP  demonstration  site,  hillshade  Hillshade  image,  alternate  settings  showing  less  dark  grey 
image,  ArcGIS  default  settings.  The  default  settings  are  area  but  also  less  contrast  on  the  craters, 
too  dark  in  some  areas  to  reveal  the  ground  surface. 


Hillshade  image,  “neutral”  hillshade  settings.  Hillshade  image,  contrast  boosted,  highlighting  craters. 

Implications  for  Munitions  Sites 

■  The  charts  presented  above  show  that  the  accuracy  of  lidar  points  varies  primarily  with 
flight  height  and  distance  from  nadir.  Lidar  acquisitions  at  the  ESTCP  demonstration 
sites  were  conducted  from  300  to  1,000  m,  where  instrument  error  is  relatively  low. 

Lidar  collected  at  these  sites  met  their  contracted  accuracy  specifications,  and  subsequent 
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acquisitions  in  this  range  of  flight  height  should  successfully  meet  similar  accuracy 
specifications.  However,  where  accuracy  is  especially  important,  contract  specifications 
may  be  adjusted  to  require  the  vendor  to  fly  lower  and/or  use  a  narrower  field  of  view. 

■  Positional  error  in  lidar  data  will  be  higher  on  sites  with  steep  terrain.  Where  appropriate, 
lidar  accuracy  can  be  assessed  independently  for  major  terrain  types. 

■  Surveyed  points  used  for  calibration  and  quality  control  of  lidar  must  be  from  a  source  of 
higher  quality  than  the  lidar  data  itself.  Survey  errors  in  control  and  calibration  points 
can  lead  to  incorrect  assessment  of  data  quality  or  errors  in  the  entire  lidar  data  set. 

■  Feature  detection  can  be  i  by  creating  surface  models  using  the  smallest  cell  size  justified 
by  the  density  of  lidar  returns.  Users  should  experiment  with  display  methods. 

6.1.5  Estimating  Confidence  Levels  for  Feature  Detection  Under  Vegetation 


Regardless  of  the  overall  point  density  achieved,  lidar  point  density  will  always  vary  somewhat 
with  terrain  and  will  be  lower  under  vegetation.  These  variation  in  lidar  density  will  affect  the 
confidence  levels  for  detecting  ground  features  over  different  parts  of  the  site.  One  approach  to 
determining  confidence  levels  for  feature  detection  is  to  map  the  density  of  lidar  ground  returns 
(Figure  28).  Areas  with  no  returns  would  be  designated  as  areas  where  lidar  would  not  detect 
ground  features,  with  confidence  levels  increasing  with  the  number  of  returns. 

Figure  28:  Preliminary  Confidence  Levels  for  Ground  Features 


Former  Camp  Beale  ESTCP  demonstration  site,  point  density  map  based  on  reclassified  ground  points.  The  legend  shows 
ground  points  per  square  meter. 
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6.2  ACTIVITY  2:  EVALUATE  CURRENT  SOFTWARE  PACKAGES 

URS  examined  a  list  of  computer  programs  that  manipulate  three-dimensional  data.  Using  the 
criteria  listed  in  Section  5.1.2,  seventeen  software  programs  were  selected  for  further  evaluation 
(Table  5). 

Software  vendors  were  contacted  and  asked  to  participate.  Of  these,  only  four  provided  analysis 
results.  URS  reviewed  the  analysis  results  and  discussed  the  findings  with  the  vendor  staff.  In 
addition  to  these  software  packages,  URS  reviewed  several  free  software  products  for  viewing 
lidar  points,  and  experimented  with  a  tentative  approach  to  crater  detection  developed  by  URS. 
These  software  packages  were: 

■  LP  360  (and  other  free  lidar  viewers) 

■  Lidar  analyst 

-  TLiD 

■  QT  Modeler 

-  HTFC 


Table  5:  Software  Packages  Selected  for  Further  Investigation 


Software  Name 

Vendor 

AcuScene 

AcuSoft 

Amber  iQ 

AmberCore  Software 

CARTERRA  Analyst 

Space  Imaging 

Creator  Pro 

MultiGen-Paradigm 

GeoACE 

US  Army  Corps  of  Engineers 

Geospatial  Analyst  Suite 

Visual  Teaming  Systems 

HTFC 

Sandia  National  Tabs 

Fidar  Analyst 

Visual  Teaming  Systems 

FidarEngine 

PCI  Geomatics 

Fidar  Works 

Technology  Service  Corporation 

FIDAR  XFR8R 

Airborne  1 

Poly  Works 

InnovMetric 

QT  Modeler 

Applied  Imagery 

Stratos98 

RockWare 

TerraScan 

Terrasolid  Oy 

TFiD 

Tiltan  Systems  Engineering 

WinATFAS 

KFT  Associates 

These  software  products  all  offer  interesting  capabilities  for  viewing  and  analyzing  lidar  data, 
and  could  be  useful  to  Government  staff  using  lidar  (Figure  29).  The  free  lidar  viewers,  although 
offering  more  limited  functionality,  also  are  appropriate  tools  for  any  regular  user  of  lidar  data. 


Page  53  of  71 


Project  Number  07  E-MM2-012/  MM-0737 
Development  of  Parameters  for  the  Collection  and 
Analysis  of  Lidar  at  Military  Munitions  Sites 
Draft  Final  Report 


However,  with  the  exception  of  the  Sandia  National  Laboratory  HTFC  software,  none  provided 
significant  potential  for  automated  detection  of  munitions-related  features  such  as  craters  or 
bombing  targets,  nor  did  they  provide  scripting  languages  for  the  development  of  such 
capabilities  by  users. 

The  Sandia  HTFC  application  was  funded  by  ESTCP  specifically  for  researching  automated 
detection  of  munitions-related  features  from  lidar  data  (Figure  30).  Sandia  investigated  several 
algorithms  for  crater  detection,  embedding  these  in  a  user  interface  constructed  in  Visual  Basic. 
Preliminary  results  were  promising,  as  detailed  in  Sandia’s  report  to  ESTCP  (Roberts  and 
McKenna  2006). 

At  the  May  2009  In-Progress  Review  meeting,  URS  and  the  US  Army  Corps  of  Engineers 
recommended  that  no  further  software  testing  be  undertaken,  and  that  any  additional  efforts  be 
directed  to  further  development  of  the  Sandia  HTFC  application.  This  recommendation  was 
based  primarily  on  the  fact  that  none  of  the  commercially-available  software  applications 
reviewed  can  detect  craters  or  other  munitions  features,  and  none  offered  scripting  support  that 
might  allow  such  capability  to  be  developed  by  users.  Secondly,  the  HTFC  application  shows 
promise  and  could  be  developed  reasonably  rapidly  should  ESTCP  wish  to  do  so.  If  desired,  the 
URS  elevation  difference  approach  could  be  tested  by  Sandia  for  possible  inclusion  in  their 
application. 

Figure  29:  Software  for  Manipulating  Lidar  Data 


LP  360.  Several  free  lidar  viewers  are  available.  None  extract 
ground  features  but  all  display  point  clouds  in  ways  that  GIS 
products  do  not.  The  illustration  above  is  from  LP  360,  an 
ArcGIS  plug-in  that  works  directly  with  the  common  .las  file 
format  and  converts  ascii  format  to  .las. 


TLiD.  TLiD  provides  a  robust  set  of  analysis  tools  for  both 
surfaces  and  point  clouds.  The  software  is  an  ArcGIS  plug-in, 
works  directly  with  .las-format  files,  and  identifies  and  extracts 
trees  and  buildings.  It  does  not  extract  munitions  features 
without  additional  programming,  and  no  scripting  capability  is 
available. 
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QT  Modeler.  QT  Modeler  provides  a  robust  set  of  analysis 
tools  for  both  surfaces  and  point  clouds.  The  software  is  an 
ArcGIS  plug-in  and  works  directly  with  .las-format  files.  It 
does  not  extract  munitions  features  without  additional 
programming,  and  no  scripting  capability  is  available. 


URS  AZ  Approach.  URS  tested  a  crater  identification 
approach  based  on  detecting  local  variations  in  ground  surface 
elevation.  The  approach  showed  some  promise  in  relatively 
flat  terrain. 
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HTFC.  Examples  including  test  data,  results  of  manual  vs.  automated  output,  and  a  portion  of  the  user  interface. 


6.3  ACTIVITY  3:  SUMMARIZE  LESSONS  LEARNED  FROM  THE  ESTCP  WAA 
PILOT  PROGRAM 

Appendix  D  contains  the  final  guidance  document  presented  to  ESTCP  in  January,  2010. 
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7.0  PERFORMANCE  ASSESSMENT 

The  performance  objectives  for  this  project  were  originally  established  in  the  Demonstration 
Plan  (URS  2008b).  Performance  objectives  were  revised  following  the  In-Progress  Review 
meetings  of  February  and  October  2008,  to  reflect  the  addition  of  ESTCP’s  requests  for  the  two 
white  papers  (Appendices  B  and  C). 

Actual  performance  compared  to  the  performance  objectives  is  summarized  in  Table  6.  In 
summary,  all  of  the  performance  objectives  were  accomplished  except  in  the  following  two 
cases: 


■  Vegetation  size  classes  were  not  mapped,  since  at  the  Former  Camp  Beale  site,  size 
classes  had  no  clear  boundaries  and  could  not  be  mapped  in  a  meaningful  manner 

■  Correlation  for  grass  heights  between  lidar  and  field  measures  could  not  be  established 

In  addition,  URS  and  USACE  recommended  that  Activity  2,  the  software  evaluation,  be  ended 
after  the  first  round  of  evaluation. 

This  section  details  the  principal  findings  of  this  demonstration. 

Activity  1  -  Systematically  investigate  vegetation  effects 

Lidar  can  perform  well  at  vegetated  sites,  though  vegetation  will  cause  some  decrease  of  lidar 
density  and  thus  lower  confidence  in  feature  detection. 

Lidar  measures  of  vegetation  height  and  density  correlate  reasonably  well  with  standard 
measures. 

Lowering  the  lidar  data  density  affected  the  feature  detection  under  vegetation.  Significant  loss 
of  feature  detection  began  to  appear  6.9  pts/m2  and  3.4  pts/m2,  with  over  two-thirds  of  the 
features  not  visible  at  the  lowest  density  level  tested  (0.8  pts/m2). 

At  lower  densities,  object  detection  rates  were  lower,  but  the  effect  is  not  completely  predictable: 
some  features  faded  gradually  as  lidar  density  lowered,  while  others  did  not  disappear  at  all. 
These  non-linear  relationships  are  to  be  expected  since  any  density  of  lidar  points  will  show  the 
features  that  happen  to  receive  the  laser  reflections.  There  was  some  correlation  between  the  rate 
of  degradation  and  the  size  of  the  feature;  that  is,  larger  features  tended  to  disappear  more 
slowly.  However,  this  correlation  was  not  strong.  Lidar  data  at  the  higher  densities  detected  all 
of  the  features  at  the  test  plots,  as  verified  by  the  field  visit. 

The  implications  of  these  non-linear  relationships  between  lidar  data  density  and  feature 
detection  include: 

■  There  is  no  lidar  density  at  which  no  features  will  be  detected.  Rather,  lower-density 
lidar  will  result  in  a  lower  degree  of  confidence. 

■  Lidar  density  will  have  a  point  of  diminishing  returns,  after  which  additional  lidar  points 
will  not  reveal  additional  features. 
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Table  6:  Performance  Objectives:  Desired  and  Actual  Results 


Performance 

Objective 

Metric 

Action  to  Achieve 
Metric 

Sampling 
Frequency 
or  Timing 

Desired  Result 

Actual  Results 

Activity  1:  Systematically  Investigate  Veg 

etation  Effects  -  Field  Data  Collection 

Delineate  vegetation 
classes  using  photos 
and  lidar  data 

All  major  vegetation  size 
classes  identified 

Delineation  from 
orthophotos  followed  by 
QA/QC  review  by 
qualified  staff 

Once  at  start 
of  project 

100%  identification  of  all 
major  vegetation  classes 

Partially  accomplished. 
Vegetation  heights  were 
calculated  for  the  study 
area.  Vegetation  size 
classes  had  no  clear 
boundaries  and  could  not  be 
delineated  meaningfully. 

Calculate  lidar  ground 
and  vegetation  point 
percentages 

Percentages  accurately 
calculated  on  a  per  square 
meter  basis  using  a  3  m 
cell  size  to  show  variation 
across  the  study  area 

Use  of  standard  GIS 
calculation  methods 
followed  by  QA/QC 
review 

Once  at  start 
of  project 

100%  of  study  area 
characterized 

Accomplished.  Percent  of 
ground  and  vegetation 
returns  was  calculated  based 
on  vendor  classification  and 
revised  classification. 

Calculate  lidar  data 
density  variability 

Density  of  ground  lidar 
points  and  all  lidar  points 
accurately  calculated  on  a 
per-meter  basis  to  show 
variation  across  the  study 
area 

Use  of  standard  GIS 
calculation  methods 
followed  by  QA/QC 
review 

Once  at  start 
of  project 

100%  of  study  area 
characterized 

Accomplished.  Densities 
for  ground  points  and  all 
points  were  calculated  on  a 
per-meter  basis. 

Select  representative 
field  plots 

Representative  study 
areas  selected 

Select  at  least  50  plot 
locations,  with  plots 
distributed  in  all 
vegetation  classes 

Once  at  start 
of  project 

Plots  placed  in  all 
vegetation  classes 

Accomplished 

Collect  field 
vegetation  density 
data 

Collect  accurate  data  on 
vegetation  conditions  at 
each  plot 

Standard  forestry 
methods  for  stand  exams 
for  deteriorating 
vegetation  density 
incorporated  into  SOP, 
followed  by  QA/QC 
review 

Once 

All  field  data  collected  to 
follow  SOP 

Accomplished 
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Performance 

Objective 

Metric 

Action  to  Achieve 
Metric 

Sampling 
Frequency 
or  Timing 

Desired  Result 

Actual  Results 

Collect  field  ground 
feature  data 

Collect  accurate  data  on 
ground  features  at  each 
plot 

Incorporate  field 
procedures  into  SOP, 
followed  by  QA/QC 
review 

Once 

All  field  data  to  follow  SOP 

Accomplished 

Activity  1:  Systematically  Investigate  Vegetation  Effects  -  Data  Analysis 

Step  1:  Classify  Vegetation 

Lidar  ground  point 
density  vs.  field 
vegetation  density 

Difference  between 
vegetation  (crown) 
density  as  measured  by 
lidar  and  field  methods 

Standard  GIS  analysis 
methods,  followed  by 
QA/QC  review 

Once 

Correlation  should  be 
highest  in  plots  with  a 
single,  simple  vegetation 
class 

Accomplished 

Lidar  vegetation 
heights  vs.  field 
vegetation  heights 

Difference  between 
vegetation  heights  as 
measured  by  lidar  and 
field  methods 

Comparison  of  lidar  and 
field  data 

Once 

Correlation  should  be 
within  5  feet  for  isolated 
trees 

Partially  accomplished. 
Correlation  for  tree  heights 
was  within  desired  results 
except  for  occasional 
outliers.  Correlation  for 
grass  height  was  poor. 

Step  2:  Identify  Ground  Features 

Field  ID  of  features 
vs.  ID  of  features 
from  lidar 

Percentage  detection  and 
false  alarm  rate  for 
features  detected  using 
lidar  vs.  field  observation 

Visual  observation  of 
lidar  data  followed  by 
comparison  of  results  of 
lidar  and  field 
observations 

Once 

Correlation  should  be  100% 
for  features  over  1  m  in  size 
in  plots  with  no  covering 
vegetation 

Accomplished 

Effects  of  lidar  point 
density  on  feature 
detection 

Number  of  features 
visible  at  successively 
lower  point  densities 

Artificially  lower  point 
density  through  a  series 
of  levels,  create  surface 
models  and  examine  the 
detection  of  features 
visible  at  the  highest 
density 

Once 

Determine  the  lidar  data 
density  where  detection 
falls  off  substantially 

Accomplished 
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Performance 

Objective 

Metric 

Action  to  Achieve 
Metric 

Sampling 
Frequency 
or  Timing 

Desired  Result 

Actual  Results 

Step  3:  Produce  White  Paper  on  Lidar  Point  Classification  Methods 

Determine  the 
percentage  of  lidar 
points  misclassified 
as  non-ground  returns 

Percentages  of  points 
classified  as  ground  or 
non-ground  vs.  actual 
point  count  in  selected 
vegetation-free  locations. 

Analysis  in  GIS  to 
determine  percentage 
values  at  selected  test 
areas  to  tally  actual  point 
classification  distribution 

Once 

Understanding  of  the  effects 
of  classification  methods  on 
percentages  of  points 
classified  as  non-ground  on 
vegetation-free  surfaces 

Accomplished 

Determine  the  source 
of  misclassification  of 
lidar  points  as  non¬ 
ground  returns 

Clear  understanding  of 
the  source  of 
misclassification. 

Conversations  with  data 
classification  staff  and 
examination  of  software 
manuals  for  classification 
software 

Once 

Clear  understanding  of  the 
source  of  misclassification 

Accomplished 

Examine  the  potential 
benefit  of 
reclassification  of 
lidar  points  to  feature 
detection 

Approximate  number  and 
size  of  features  visible 
after  reclassification 

Create  new  surface 
models  using  reclassified 
points  and  examine 

Once 

Estimate  of  the  potential 
benefit  of  point 
reclassification 

Accomplished 

White  paper  technical 
accuracy 

Point  classification  white 
paper  accurate 

Peer  review  by 
specialists  from  industry 
and  universities 

Once 

Document  will  be 
technically  accurate 

Accomplished.  Document 
was  subject  to  technical 
peer  review  by  vendors 
familiar  with  classification 
methods  and  software. 

White  paper 
readability 

Point  classification  white 
is  well  written  and 
understandable  by  non¬ 
specialists 

Technical  editing  and 
QA/QC  review 

Once 

Document  will  be 
understandable 

Accomplished 

Step  4:  Produce  White  Paper  on  Lidar  Error 

White  paper  technical 
accuracy 

White  paper  on  lidar  error 
is  technically  accurate 

Peer  review  by 
specialists  from  industry 
and  universities 

Once 

Document  will  be 
technically  accurate 

Accomplished 
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Performance 

Objective 

Metric 

Action  to  Achieve 
Metric 

Sampling 
Frequency 
or  Timing 

Desired  Result 

Actual  Results 

White  paper 
readability 

White  paper  on  lidar  error 
is  well  written  and 
understandable  by  non¬ 
specialists 

Technical  editing  and 
QA/QC  review 

Once 

Document  will  be 
understandable 

Accomplished 

Activity  2:  Evaluate  Current  Software  Packages 

Prepare  appropriate 
lidar  data  sets  for 
subsequent  testing 

Accurate  ID  of  ground 
features  for  testing 

Visual  inspection 
followed  by  QA/QC 
review 

Once  at  start 
of  project 

100%  confidence  in  test 
features  to  be  given  to 
vendors 

Accomplished. 

ID  of  features  using 
automated  methods 

Percent  detection  and 
false  alarm  rate  of 
automated  methods  vs. 
visual  inspection  results 

Compare  results  of  visual 
inspection  and  automated 
methods 

Once  from 
each  vendor 

Percent  detection  will  be 
highest  for  isolated,  well- 
defined  features.  False 
alarm  rate  will  be  lowest  for 
relatively  flat,  smooth 
ground  surfaces. 

Partially  accomplished. 
Only  one  software  product 
reviewed  (Sandia  National 
Labs  HTFC)  was  able  to 
automate  feature  detection. 
As  this  was  an  ESTCP- 
funded  product,  percent 
detection  and  false  alarm 
rate  were  documented 
already.  Based  on  these 
initial  results,  URS  and 

US  ACE  recommended  that 
Activity  2  not  proceed  to 
further  testing,  and  that 
ESTCP  work  directly  with 
Sandia  on  further 
development. 

Activity  3:  Summarize  Lessons  Learned  from  the  ESTCP  WAA  Pilot  Program 

Usability 

Guidance  document 
applicable  to  the 
requirements  of  present 
and  anticipated  munitions 
management  programs 

Peer  review  by  DoD 
program  and  policy  staff 

Once 

Document  will  be  useable 

Pending 
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Performance 

Objective 

Metric 

Action  to  Achieve 
Metric 

Sampling 
Frequency 
or  Timing 

Desired  Result 

Actual  Results 

Technical  accuracy 

Guidance  document  is 
technically  accurate 

Peer  review  by 
specialists  from  industry 
and  universities 

Once 

Document  will  be 
technically  accurate 

Accomplished 

Readability 

Guidance  document  is 
well  written  and 
understandable  by  non¬ 
specialists 

Technical  editing  and 
QA/QC  review 

Once 

Document  will  be 
understandable 

Accomplished 

DoD  -  Department  of  Defense 
GIS  -  Geographic  Information  Systems 
ID  -  identification 
m  -  meter 

QA/QC  -  quality  assurance/quality  control 
SOP  -  standard  operating  procedure 
US  ACE  -  US  Army  Corps  of  Engineers 
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The  performance  of  lidar  can  be  improved  compared  to  standard  classification  and  display 
methods.  Once  a  sufficiently  dense  lidar  data  set  has  been  acquired,  performance  can  be 
improved  using  the  following  approaches: 

■  Point  classification  methods  can  be  adjusted  to  add  more  points  into  the  ground  surface 
model.  The  elevation  cut-off  for  adding  additional  points  probably  should  be  established 
based  on  individual  site  conditions,  but  findings  suggest  that  this  method  can  increase  the 
resolution  of  the  ground  surface  model  and  justify  the  use  of  smaller  grid  cells,  both  of 
which  have  the  potential  to  reveal  additional  ground  features  if  present. 

■  Surface  models  can  be  created  using  the  smallest  grid  cell  size  justifiable  given  the  point 
density. 

In  practice,  surface  model  development  may  be  more  conveniently  accomplished  in-house  by 
Government  end-users,  rather  than  having  the  vendor  deliver  them.  This  is  because  in-house 
development  can  allow  for  experimentation  with  alternative  methods.  However,  this  requires 
that  Government  users  have  sufficient  software  tools  and  training  to  accomplish  these  tasks. 
Creating  DEMs  and  DTMs  in-house  can  be  challenging,  especially  with  high-density  data  sets 
where  the  number  of  points  is  very  large.  In  such  cases  it  may  be  appropriate  to  work  with 
sample  data  sets  to  determine,  in  consultation  with  vendors,  the  appropriate  specifications  for 
these  products,  and  then  request  that  the  vendor  create  and  deliver  the  final  products. 

For  lidar  points,  controlling  error  is  primarily  a  function  of  standard  quality  control  methods,  and 
the  accuracy  of  the  lidar  points  is  best  assured  by  establishing  and  adhering  to  appropriate 
contract  specifications  during  data  acquisition.  Specifications  for  lidar  accuracy  can  be 
independently  verified  by  end  users. 

Confidence  levels  for  feature  detection  can  be  mapped.  As  shown  in  Figure  28,  it  is  possible  to 
map  the  number  of  ground  returns  per  square  meter,  using  either  the  vendor’s  classification  or  a 
re-classified  data  set.  Site  managers  and  regulators  can  use  these  maps  to  classify  areas  where 
insufficient  lidar  points  reached  the  ground  surface  to  characterize  features  of  a  given  size. 

Fidar  vendors  should  always  be  asked  to  deliver  the  full  lidar  point  set.  Because  lidar  data  sets 
are  very  large,  many  vendors  only  deliver  derived  products  such  as  ground  surface  models 
(DEMs)  and  all-points  models  (DTMs).  The  results  of  this  investigation  underscore  the 
importance  of  receiving  the  entire  lidar  point  data  set  for  all  lidar  investigations.  Using  this  data, 
Government  land  managers  can  evaluate  the  approaches  to  point  classification  and  to  surface 
model  creation  used  by  the  vendor,  and  make  appropriate  adjustments. 

Activity  2  -  Evaluate  current  software  packages 

Current  software  products  offer  many  useful  tools  for  manipulating  lidar  data,  but  of  those 
reviewed,  only  the  HTFC  application  by  Sandia  National  Faboratories  was  able  to  detect  craters. 
Users  of  lidar  at  munitions  sites  could  consider  funding  the  completion  and  distribution  of  the 
HTFC  application. 
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Lidar  software  is  being  improved  continuously.  Vendors  should  be  contacted  periodically  to 
review  new  developments.  Government  offices  that  use  lidar  data  also  should  be  contacted  to 
determine  which  software  products  are  currently  in  use  and  whether  they  are  performing  well. 

Activity  3  -  Summarize  lessons  learned  from  the  ESTCP  WAA  Pilot  Program 

A  revised  draft  guidance  document  (Appendix  D)  was  delivered  to  ESTCP  and  technical  peer 
reviewers  in  July  2009.  The  final  version  incorporating  peer  review  is  submitted  with  this  report. 
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8.0  COST  ASSESSMENT 

8.1  COST  MODEL 

Cost  models  for  lidar  and  orthophotos  are  presented  in  the  final  reports  for  the  ESTCP 
demonstration  sites  (URS  2007  and  2008a)  and  for  the  ESTCP  WAA  Pilot  Program  (Nelson  et 
al.  2008).  The  results  of  this  demonstration  led  to  small  changes  in  the  cost  models,  as  described 
in  Section  8.2. 

8.2  COST  DRIVERS 

Cost  drivers  for  lidar  and  orthophoto  acquisition  are  discussed  in  detail  in  the  final  reports  for  the 
ESTCP  demonstration  sites  and  for  the  Pilot  Program  as  a  whole.  Additional  conclusions  from 
this  investigation  include  the  following: 

■  Cost  for  data  acquisition  at  vegetated  sites  should  not  be  significantly  higher  than  for 
non-vegetated  sites,  except  in  very  exceptional  circumstances.  Lidar  at  vegetated  sites 
should  be  collected  at  a  higher  density;  however,  with  the  introduction  of  higher-speed 
sensors  (up  to  250  kHz  pulse  repetition  rate),  appropriate  data  densities  for  vegetated  sites 
will  rarely  require  additional  flight  lines.  Costs  for  processing  the  additional  lidar  data  at 
vegetated  sites  may  be  somewhat  higher  based  on  the  need  for  editing  of  the 
classification  results  by  skilled  operators. 

■  At  vegetated  sites,  the  cost  of  orthophoto  acquisition  could  be  lower  since  orthophotos 
could  be  acquired  with  larger  pixel  sizes  or  omitted  in  favor  of  using  pre-existing 
orthophotos. 

■  There  should  be  little  additional  cost  for  changing  data  classification  specifications  to 
return  more  points  to  the  ground  surface  model.  At  most,  the  vendor  would  charge  for 
some  initial  tests  of  alternate  processing  methods,  or  for  establishing  an  additional 
category  for  low  lidar  points.  These  additional  costs  should  not  be  high. 

■  This  project  showed  that  feature  detection  rates  did  not  decline  substantially  between  the 
highest  data  density  tested  (13.8  pts/m2)  and  half  of  that  density  (6.9pts/m2).  This 
suggests  that  it  is  unnecessary  to  collect  extremely  high  lidar  data  densities.  This  may 
result  in  some  cost  savings. 

8.3  COST  BENEFIT 

The  final  reports  for  the  individual  ESTCP  demonstration  sites  (URS  2007  and  2008a)  and  for 
the  ESTCP  WAA  Pilot  Program  (Nelson  et  al.  2008)  showed  that  lidar  was  a  cost-effective 
addition  to  the  range  of  WAA  technologies.  As  summarized  in  Section  7,  the  current 
demonstration  examined  several  potential  ways  to  increase  the  effectiveness  of  lidar  data.  While 
most  of  these  would  not  change  the  overall  cost  of  using  these  technologies,  each  would  increase 
the  benefit  to  cost  ratio. 
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9.0  IMPLEMENTATION  ISSUES 

As  an  airborne  technology,  implementation  of  lidar  does  not  present  significant  regulatory 
challenges.  Lidar  and  orthophotography  rely  on  commercial  off-the-shelf  equipment,  and  there 
is  a  network  of  qualified  vendors.  There  are  few  issues  with  equipment  availability  or  skilled 
operators.  Procurement  issues  discussed  in  this  demonstration  include  establishing  appropriate 
contract  specifications,  especially  those  related  to  point  density  specifications  and  point 
classification  methods. 
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11.0  GLOSSARY 

accuracy:  The  closeness  of  an  estimated  value  to  a  standard  or  accepted  value  of  a  particular 
quantity. 

anomaly:  A  geophysical  signal  from  a  detected  subsurface  object  above  geological  background. 

artifacts:  In  lidar,  detectable  surface  remnants  of  buildings,  trees,  towers,  telephone  poles  or 
other  elevated  features  in  a  bare-earth  elevation  model.  Also,  detectable  artificial  anomalies  that 
are  introduced  to  a  surface  model  via  system-specific  collections  or  processing  techniques. 

bathymetry:  The  measurement  and  study  of  water  depths. 

calibration:  The  process  of  identifying  and  correcting  for  systematic  errors  in  hardware, 
software,  or  procedures. 

conceptual  site  model  (CSM):  A  description  of  site  conditions  that  coveys  what  is  known  or 
suspected  about  the  sources,  releases  and  release  mechanisms,  contaminant  fate  and  transport, 
exposure  pathways,  potential  receptors,  and  risks. 

contours:  Lines  of  equal  elevation  on  a  surface.  An  imaginary  line  on  the  ground,  all  points  of 
which  are  at  the  same  elevation  above  or  below  a  specified  reference  surface. 

decimate:  In  the  context  of  lidar,  artificially  lowering  the  density  of  the  lidar  data  points  in  a 
manner  that  simulates  the  action  of  a  slower  lidar  data  sensor. 

digital  elevation  model  (DEM):  A  generic  term  for  digital  topographic  and/or  bathymetric  data 
in  all  its  various  forms,  but  most  often  bare  earth  elevations  at  regularly  spaced  intervals  in  x  and 
y  directions.  Regularly  spaced  elevation  data  are  easily  and  efficiently  processed  in  a  variety  of 
computer  uses. 

digital  terrain  model  (DTM):  Similar  to  DEMs,  but  they  may  incorporate  the  elevation  of 
significant  topographic  features  on  the  land  and  mass  points  and  break  lines  that  are  irregularly 
spaced  to  better  characterize  the  true  shape  of  the  bare  earth  terrain. 

digital  surface  model  (DSM):  Similar  to  DEMs  or  DTMs,  except  they  may  depict  the  elevations 
of  the  top  surfaces  of  buildings,  trees,  towers,  and  other  features  elevated  above  the  bare  earth. 

electromagnetic  induction  (EMI):  The  physical  process  by  which  a  secondary  electromagnetic 
field  is  induced  in  an  object  by  a  primary  electromagnetic  field  source. 

Geographic  Information  System  (GIS):  A  system  of  spatially  referenced  information,  including 
computer  programs  that  store,  manipulate,  analyze,  and  display  spatial  data. 

Global  Positioning  System  (GPS):  Technology  that  computes  the  three-dimensional  position  of 
an  object  in  space,  for  example  the  lidar  sensor,  using  signals  from  at  least  four  orbiting 
navigation  satellites. 

hillshade:  A  function  used  to  create  an  illuminated  representation  of  a  surface,  using  a 
hypothetical  light  source,  to  enhance  visualization  effects. 

horizontal  accuracy:  The  positional  accuracy  of  a  dataset  with  respect  to  a  horizontal  datum. 
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Inertial  Measurement  Unit  (IMU):  Technology  that  uses  gyroscopes  and  accelerometers  to 
compute  the  roll,  pitch,  and  heading  of  a  moving  object,  for  example  a  lidar  sensor. 

Inertial  Navigation  System  (INS):  A  navigation  aid  that  uses  a  computer  and  motion  sensors 
(the  IMU)  to  continuously  calculate  via  dead  reckoning  the  position,  orientation,  and  velocity  of 
a  moving  object  without  the  need  for  external  references. 

light  detection  and  ranging  (lidar):  An  instrument  system  that  measures  distance  to  a  reflecting 
object  by  emitting  timed  pulses  of  laser  light  and  measuring  the  time  between  emission  and 
reception  of  reflected  pulses.  The  measured  time  interval  is  converted  to  distance. 

magnetometry:  The  technique  of  measuring  and  mapping  patterns  of  the  earth’s  magnetic  field 
as  modified  by  geology  or  ferrous  objects. 

mrad:  mRad  is  short  for  milliradians,  a  measure  of  the  angle  at  which  the  laser  beam  expands 
with  distance  from  its  origin.  A  divergance  of  1  mRad  would  be  roughly  equal  to  the  beam 
expanding  1mm  for  every  1  meter  it  travels. 

munitions  and  explosives  of  concern  (MEC):  Specific  categories  of  military  munitions  that 
may  pose  unique  explosives  safety  risk:  unexploded  ordnance,  discarded  military  munitions,  or 
munitions  constituents  such  as  TNT  or  RDX  present  in  high  enough  concentrations  to  pose  an 
explosive  hazard. 

nanometer:  A  unit  of  length  equal  to  one  billionth  of  a  meter. 
ordnance:  Weapons  of  all  kinds. 

orthophotograph:  A  digital  aerial  photograph  that  has  been  geometrically  corrected  for 
topographic  relief,  lens  distortion,  and  camera  tilt. 

orthorectification:  The  process  by  which  the  geometric  distortions  of  an  image  are  modeled  and 
accounted  for. 

positional  accuracy:  The  accuracy  of  the  position  of  features,  including  horizontal  and/or 
vertical  positions. 

redundant  array  of  independent  disks ”  (RAID)  device:  A  device  containing  multiple  hard 
drives,  allowing  computer  users  to  achieve  higher  levels  of  storage  reliability  from  low-cost  and 
less  reliable  PC-class  disk-drive  components,  through  the  technique  of  arranging  the  devices  into 
redundant  arrays. 

root  mean  square  error  (RMSE):  An  accuracy  assessment  for  measured  data  (e.g.,  lidar) 
calculated  by  taking  the  square  root  of  the  average  of  the  set  of  squared  differences  between 
dataset  values  (i.e.,  lidar  derived  elevations  versus  field  surveyed  elevations). 

triangulated  irregular  network  (TIN):  A  set  of  adjacent,  non-overlapping  triangles  computed 
from  irregularly  spaced  points  with  x/y  coordinates  and  z-values. 

unexploded  ordnance:  Military  munitions  that  have  been  primed,  fused,  armed,  or  otherwise 
prepared  for  action  and  that  have  been  fired,  dropped,  launched,  or  placed  in  a  manner 
constituting  a  hazard  to  operations,  installations,  or  personnel;  which  remain  unexploded. 


Page  70  of  71 


Project  Number  07  E-MM2-012/  MM-0737 
Development  of  Parameters  for  the  Collection  and 
Analysis  of  Lidar  at  Military  Munitions  Sites 
Draft  Final  Report 


vegetation  removal:  In  lidar,  the  correction  of  reflective  surface  elevations  so  as  to  depict  the 
elevation  of  the  bare  earth  terrain  beneath  the  vegetation. 

vertical  accuracy:  The  measure  of  the  positional  accuracy  of  a  dataset  with  respect  to  a 
specified  vertical  datum. 

wide  area  assessment:  Rapid  assessment  of  large  tracts  of  potentially  contaminated  land  to 
identify  those  areas  with  concentrated  military  munitions  that  require  detailed  characterization. 
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1  Introduction  and  Objectives 


1 .1  Introduction  -  Lidar  in  Wide  Area  Assessment 

Many  millions  of  acres  of  Department  of  Defense  (DoD)  lands  are  potentially 
contaminated  with  military  munitions  or  their  components.  On  the  majority  of  these  sites, 
munitions  are  concentrated  in  specific  ranges  and  training  areas.  Locating  the  site  of 
contamination  can  be  difficult,  in  part  because  historical  records  are  often  incomplete  or 
inaccurate. 

Between  2005  and  2007,  the  Environmental  Security  Technology  Certification  Program 
(ESTCP)  conducted  a  pilot  program  to  test  the  effectiveness  of  a  multi-technology 
approach  to  unexploded  ordnance/munitions  and  explosives  of  concern  (UXO/MEC) 
wide  area  assessment  (WAA).  The  program  included  the  use  of  light  detection  and 
ranging  (lidar),  orthophotography,  helicopter  magnetometry,  towed-array  magnetometry, 
and  statistically-based  transect  design.  The  first  phase  of  this  program  was  carried  out 
at  three  desert  sites  containing  little  or  no  vegetation  and  few  non-military  land  uses:  the 
Pueblo  Precision  Bombing  Range  site  near  Pueblo,  Colorado;  Kirtland  Air  Force  Base 
Precision  Bombing  Range  site  near  Albuquerque,  New  Mexico;  and  Victorville 
Demolition  Bombing  Range  near  Victorville,  California.  Subsequently,  a  second  phase 
of  the  pilot  program  was  added,  including  two  new  sites:  the  Former  Camp  Beale  site 
near  Marysville,  California,  and  the  Toussaint  River  site  near  Lake  Erie.  The  Former 
Camp  Beale  site  has  more  varied  vegetation  cover  and  land  use  types;  the  Toussaint 
River  site  is  a  shallow-water  site  and  only  a  limited  amount  of  land-based  lidar  was 
acquired. 

At  the  three  desert  sites  and  the  Former  Camp  Beale  site,  lidar  was  successfully  used  to 
identify  munitions  response  sites,  including  bombing  targets,  berms,  and  firing  points 
that  were  not  detected  using  other  technologies.  At  all  four  sites,  lidar  was  used  to 
detect  small  ground  features  such  as  potential  craters.  In  many  cases  both  the  targets 
and  the  small  ground  features  were  highly  eroded  and  not  visible  to  ground  crews. 

At  all  four  sites,  the  lidar  investigation  was  able  to  supplement  or  correct  the  initial 
conceptual  site  model  based  on  historical  records  review.  This  was  accomplished 
through  the  discovery  of  preciously  unknown  targets,  the  correction  of  incorrect  locations 
of  known  targets,  and  the  definition  of  boundaries  for  known  targets.  Lidar  also  provided 
useful  input  to  subsequent  phases  of  site  investigation,  including  information  on  slope 
and  vegetation  cover. 

1.2  Problem  Statement  and  Objectives 

Error  in  the  collection,  processing,  and  interpretation  of  lidar  data  can  lead  to  less 
accurate  site  characterization,  with  potential  for  “missed”  ground  features,  difficulty  in 
integrating  lidar  with  other  spatial  data,  and  incorrect  assessment  of  confidence  levels  in 
the  data.  An  understanding  of  error  in  lidar  data  can  contribute  to  appropriate 
expectations  for  the  technology,  and  to  the  development  of  appropriate  contract 
specifications. 

Understanding  lidar  error  is  especially  important  since  error  in  lidar  data  arises  from 
different  factors  than  error  in  magnetometry  or  electromagnetic  induction  (EMI),  the  most 
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common  technologies  used  in  geophysical  investigations  of  munitions  sites. 
Magnetometry  and  EMI  are  used  to  directly  detect  and  map  magnetic  anomalies  that 
may  result  from  MEC,  and  each  anomaly  is  reported  and  located  individually.  Error  is 
understood  as  the  discrepancy  between  the  reported  location  of  the  magnetic  anomaly 
and  the  true  location  of  the  object. 

Lidar,  by  contrast,  uses  large  arrays  of  laser  reflections  to  model  surfaces.  The  location 
of  surface  features  (potential  craters,  for  instance)  is  then  inferred  from  characteristics  of 
the  entire  surface.  The  accuracy  of  the  location  of  features  in  the  modeled  surface  rests 
on  the  accuracy  and  precision  of  the  individual  lidar  points,  but  is  influenced  by  other 
factors  such  as  the  density  of  the  laser  returns,  the  characteristics  of  the  terrain,  and  the 
methods  used  to  create  the  surface  model.  These  factors  can  interact  in  complex  ways. 

This  white  paper  presents  a  description  of  sources  and  magnitudes  of  error  in  lidar  data, 
and  the  potential  implications  of  such  error  for  the  use  of  lidar  at  military  munitions  sites. 
The  objective  of  the  paper  is  not  to  present  the  full  range  of  scientific  research  on  lidar 
error.  Rather,  it  is  intended  as  a  guide  for  Government  land  managers  who  may  acquire 
lidar.  It  identifies  major  sources  of  error,  describes  their  implications  for  the  use  of  lidar 
at  munitions  sites,  and  presents  approaches  for  minimizing  their  impact. 

The  paper  discusses: 

■  Factors  affecting  the  accuracy  and  precision  of  the  lidar  points 

■  Error  in  GPS 

■  Factors  affecting  the  accuracy  of  digital  surface  models 

■  Published  map  accuracy  standards  in  common  use 

Quantitative  measures  of  error  are  available  primarily  in  the  context  of  instrument  error 
and  terrain  error.  Factors  affecting  the  digital  surface  models  are  important  but  more 
difficult  to  quantify. 

This  paper  is  based  on  published  research  and  equipment  specifications,  along  with 
interviews  with  lidar  vendors  and  software  providers.  Examples  are  taken  from  lidar 
collected  at  ESTCP  demonstration  sites  and  other  lidar  data  in  the  public  domain.  No 
original  research  was  conducted  for  this  paper 
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2  Accuracy  and  Precision  of  Lidar  Points 


Error  in  the  use  of  lidar  derives  initially  from  the  accuracy  and  precision  of  the  individual 
lidar  points. 

2.1  Lidar  Accuracy  Specifications 

Lidar  vendors  guarantee  the  accuracy  of  lidar  points  as  a  part  of  their  contract 
documents,  and  vendors  quote  very  similar  guarantees  across  the  industry.  A  typical 
accuracy  specification  quoted  is  15  cm  vertical  and  50-100  cm  horizontal.1  Some 
vendors  provide  a  more  detailed  specification  such  as  the  following:2.  (All  values  are  at 
95%  (two  sigma  or  two  standard  deviations)) 

■  Vertical:  15  centimeters  (cm)  hard  surfaces  and  open  regular  terrain,  25  cm 
soft/vegetated  surfaces,  flat  to  rolling  terrain,  30-50  cm  soft/vegetated  surfaces, 
hilly  terrain 

■  Horizontal:  50-75  cm  in  all  but  extremely  hilly  terrain  (depends  on  flying  height 
and  beam  divergence 

Lidar  vendors  guarantee  the  accuracy  of  the  lidar  data,  that  is,  its  correspondence  to 
surveyed  control.  Accuracy,  in  contrast  to  precision,  is  the  closeness  of  an  estimated 
value  to  a  standard  or  accepted  correct  value3.  The  accuracy  value  refers  the  size  of  the 
differences  between  the  estimated  and  the  standard  value.  In  the  context  of  lidar,  the 
guaranteed  accuracy  refers  to  the  correspondence  of  surveyed  control  points  to  either 
the  individual  lidar  points  closest  to  the  surveyed  point,  or  to  the  elevation  of  the  lidar- 
derived  surface  model  at  that  point. 

2.2  Expressing  Lidar  Point  Accuracy:  Root  Mean  Square  Error 

Lidar  vendors  typically  express  their  stated  accuracies  as  root  mean  square  error 
(RMSE)  values.  In  calculating  RMSE,  the  difference  between  data  set  coordinate  values 
and  the  coordinate  values  from  an  independent  source  of  higher  accuracy  for  identical 
points  are  each  squared  and  then  averaged  over  the  sample,  after  which  the  square  root 
of  the  average  is  taken.  Since  the  errors  are  squared  before  they  are  averaged,  RMSE 
gives  a  relatively  high  weight  to  large  errors  compared  to,  for  instance,  an  accuracy 
measure  such  as  the  mean  absolute  error,  which  gives  the  same  weight  to  all  values. 
This  means  that  RMSE  is  most  useful  when  large  errors  are  particularly  undesirable. 

One  weakness  of  RMSE  is  that,  as  a  single  number,  it  does  not  capture  the  spatial 
variability  of  lidar  point  error.  As  will  be  discussed  in  this  paper,  lidar  error  can  vary  with 
terrain  and  vegetation,  and  RMSE  alone  will  not  capture  this  variation.  Useful 
supplements  to  the  use  of  RMSE  would  include  maps  of  principal  terrain  and  vegetation 
types.  Additionally,  RMSE  values  are  indications  of  the  differences  between  the  lidar 


1  See:  http://www.airborne1.com/technoloav/LiDARAccuracv.pdf. 

2  Terrapoint,  LLC,  see:  http://www.ambercore.com/files/TerrapointWhitePaper.pdf. 

3  The  discussion  in  this  section  is  adapted  from  Chapter  3  of  Digital  Elevation  Model  Technologies  and 
Applications:  the  DEM  Users  Manual  (Maune  2001). 
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points  and  survey  control,  and  do  not  in  themselves  account  for  potential  errors  in  the 
survey  control  points  themselves. 

RMSE  values  are  generally  quoted  at  the  68%  (one  sigma  or  one  standard  deviation)  or 
95%  (two  sigma  or  two  standard  deviations)  level.  This  is  because  the  error  of  some 
lidar  points  will  always  fall  outside  of  the  stated  accuracy  range. 

When  calculating  the  point  accuracy,  it  is  a  common  practice  to  eliminate  from  statistics 
all  the  points  whose  differences  exceed  a  three  sigma  value.  These  points  are 
considered  outliers  and  are  mostly  present  because  the  target  points  in  the  control  and 
the  lidar  were  wrongly  associated.  Such  outliers  are  usually  scarce,  and  the  presence  of 
a  significant  number  of  outliers  may  point  to  the  existence  of  some  systemic  source  of 
error  in  the  data. 

Formulas  for  computing  horizontal  and  vertical  RMSE  are  given  in  Attachment  A. 

2.3  Assessing  the  Accuracy  of  Lidar  Points  in  the  Field 

Horizontal  and  vertical  accuracy  of  lidar  points  are  assessed  differently.  In  measuring 
vertical  accuracy,  lidar  vendors  commonly  survey  a  variety  of  locations  in  the  study  area, 
and  compare  these  surveyed  elevations  to  lidar  elevations  at  the  same  points.  Surveyed 
points  are  commonly  established  using  static  survey  methods4.  Some  vendors  will 
supplement  static  survey  points  with  large  numbers  of  additional  points  collected  using 
kinematic  Global  Positioning  System  (GPS)  survey  methods,  collected  by  driving  along 
roads  in  the  project  area.  Other  vendors  use  small  unmanned  rover  vehicles  to  collect 
large  numbers  of  static  points. 

Surveyed  points  provide  survey-grade  data  points  of  a  higher  accuracy  than  the  lidar 
data  to  assess  the  accuracy  of  the  lidar  points.  Vendors  may  perform  vertical 
adjustments  of  the  entire  lidar  data  set  to  achieve  a  best  possible  fit  to  the  control 
points5,  in  which  case  the  residual  differences  after  this  adjustment  provide  the 
quantitative  estimate  of  the  vertical  error  of  the  lidar  data. 

Methods  and  instrumentation  chosen  to  collect  the  control  points  will  directly  influence 
the  computed  accuracy  of  lidar  points.  Since  the  accuracy  of  various  methods,  such  as 
total  station,  static,  and  kinematic  GPS,  can  vary  from  sub-centimeter  to  a  decimeter 
level  it  is  clear  that  different  methods  and  adjustments  will  produce  control  points  of 
different  accuracy,  directly  influencing  the  accuracy  estimation  of  the  lidar  points. 

Primary  control  points  should  have  an  accuracy  at  least  one  order  of  magnitude  greater 
than  the  lidar  points,  and  all  control  points  should  be  reported  with  their  method  of 
collection  and  associated  error. 


4  Vendors  generally  assume  that  these  surveyed  points  are  error-free,  which  in  reality  is  not  true.  While 
more  accurate  than  the  lidar  points,  surveyed  points  are  collected  using  techniques  and  instruments  that  are 
themselves  subject  to  error,  for  instance  total  station,  static  and  kinematic  GPS  have  accuracies  which  vary 
from  sub-centimeter  to  tens  of  centimeter.  In  practice,  the  estimated  accuracy  of  the  surveyed  points  should 
be  reported  along  with  the  points  themselves,  and  considered  in  evaluating  the  accuracy  of  the  lidar  points. 
(Personal  communication,  Terrapoint  with  Dale  Bennett,  June  2009). 

5  At  least  one  vendor  state  that  without  such  adjustment,  vertical  errors  of  the  lidar  data  would  routinely 
exceed  the  manufacturers  reported  maximum  vertical  error  of  15  cm  (Sky  Research,  2009). 
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To  evaluate  horizontal  accuracy,  vendors  compare  the  location  of  the  lidar  returns  for 
objects  in  the  study  area  whose  real-world  horizontal  locations  can  be  surveyed.  These 
may  include  target  objects  placed  with  the  surveyed  points  as  in  Figure  1 ,  or  larger 
objects.  For  example,  lidar  elevation  values  can  be  used  to  model  the  corners  of 
buildings  and  other  structures,  or  lidar  intensity  values  can  be  used  to  model  the  edges 
of  pavement  (Figure  2).  These  lidar-based  locations  can  then  be  compared  to  surveyed 
locations  for  the  same  objects. 

Figure  1:  Surveyed  Control  Points  Used  to  Assess  Vertical  Error 


Typical  surveyed  points  used  to  evaluate  vertical  and  horizontal  accuracy. 


Survey  target,  with  the-derived  point  added 
(green). 


Lidar  points  classified  by  intensity.  Blue  points  are 
reflections  from  the  target  legs.  Green  point  is  the 
derived  intersection  of  the  two  panels,  which  is 
compared  to  the  surveyed  location. 
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Figure  2:  Calibration  Objects,  Building  Edges  and  Pavement  Edges 
Used  to  Asses  Horizontal  Error 


Lidar  points  reflecting  on  and  around  a  pre-placed  rectangular  control  surface.  Lidar  points  are  color-coded  to 
show  reflections  from  the  ground  surface  (green)  and  those  at  or  above  the  known  height  of  the  flat  panel  (red). 


Lidar  elevation  values  are  used  to  model  the  edge  of  the 
building.  The  modeled  location  is  compared  to  the 
surveyed  location  -  not  to  the  location  in  the  orthophoto 
since  this  may  be  subject  to  different  sources  of  error. 


Lidar  intensity  values  are  used  to  model  the  edges 
of  pavement  (as  in  the  runway  above).  Locations  of 
pavement  edges  or  corners  are  compared  surveyed 
locations. 


Horizontal  and  vertical  accuracy  are  reported  by  vendors  using  standard  quality  control 
reports.  Table  1  shows  an  example  lidar  vertical  error  report  from  a  vendor,  displaying 


6 


ESTCP  White  Paper:  Errors  in  Lidar  Data:  Implications  for 
Investigation  of  Military  Munitions  Sites 
Project  Number  07  E-MM2-012/MM-0737 


lidar-to-control-point  vertical  error  for  eight  control  points6.  It  shows  an  error  of  just  over 
10  cm  or  0.104  meter  (m)  at  the  95%  confidence  level. 


Table  1:  Example  Lidar  Vertical  Error  Report 


Statistics 

Target 

Easting 

Northing 

Survey 

Elevation 

Lidar  Elevation 

Difference 

TAR1 

543461 .243 

3805793.824 

827.486 

827.440 

-0.046 

TAR2 

542947.381 

3807961.251 

837.761 

837.740 

-0.021 

TAR3 

543747.347 

3809712.492 

833.970 

833.930 

-0.040 

TAR4 

545277.331 

3809859.694 

797.753 

797.710 

-0.043 

TAR5 

546671 .936 

3810138.520 

846.280 

846.230 

-0.050 

TAR6 

547168.309 

3808650.964 

864.049 

864.000 

-0.049 

TAR7 

545682.049 

3807795.187 

784.720 

784.650 

-0.070 

TAR8 

546462.139 

3805793.466 

806.263 

806.180 

-0.083 

Summary 

Average  difference 

-0.050a 

Minimum  difference 

-0.083 

Maximum  difference 

-0.021 

RMSE  of  the  elevation 

+/-  0.053 

2d  RMSE  of  the  elevation  (95%) 

+/- 0.104 

Std-Dev.  of  the  elevation 

+/- 0.019 

Source:  Terra  Remote  Sensing  (2007) 

Surveyed  and  lidar  elevations  in  this  chart  are  reported  in  meters  above  the  “height  above  the  geoid”,  an  elevation  that 
roughly  coincides  with  mean  sea  level.  See  Maune  (2001 )  for  additional  discussion  of  elevation  values  in  GPS  and 
lidar. 

a.  This  value  points  to  a  residual  of  some  systemic  error  in  the  data.  Once  the  systemic  error  is  removed  from  the  data, 
the  RMSE  and  the  standard  deviation  will  converge. 


2.4  Lidar  Point  Precision 

Often  confused  with  accuracy,  precision  is  a  measure  of  the  tendency  of  a  set  of  values 
to  cluster  about  a  number  determined  by  the  set.  The  usual  measure  of  precision  is  the 
standard  deviation  or  the  standard  error.  Precision  is  distinguished  from  accuracy  in  that 
accuracy  is  a  measure  of  the  proximity  of  the  “true”  value  usually  as  established  by  an 
independent  means  of  at  least  an  order  of  magnitude  higher  in  accuracy.  (This  is  the 
basis  of  using  static  survey  techniques  to  establish  control  points  for  lidar.)  Therefore,  in 
order  to  be  highly  precise,  a  data  set  needs  only  to  conform  to  itself,  while  to  be  accurate 
the  data  set  must  conform  to  an  independently  derived  standard  (Maune  2001). 

If  lidar  precision  were  perfect,  there  would  be  no  elevation  differences,  for  instance, 
between  lidar  returns  on  a  uniformly  flat  surface.  In  practice,  precision  is  never 
completely  perfect,  and  precision  errors  appear  as  the  “corduroy”  striping  observable  on 


6  Especially  when  large  numbers  of  control  points  are  collected,  occasional  control  points  will  have  large 
differences  from  the  lidar  data.  It  is  common  practice  to  eliminate  such  data  from  statistics  as  outliers,  as 
long  as  such  outliers  are  scarce.  The  presence  of  significant  numbers  of  outliers  points  to  the  existence  of 
some  systemic  source  of  error  which  should  be  investigated.  (Personal  communication,  Terrapoint  with 
Dale  Bennett,  June  2009). 
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flat,  smooth  ground  surfaces,  or  the  “lumpiness”  observable  on  paved  parking  areas  or 
roads  (Figure  3). 


Figure  3:  Lidar  Precision  Error 


Surface  model  derived  from  lidar.  “Corduroy”  effect  on 
relatively  smooth,  flat  ground  surface  at  Former  Camp 
Beale  ESTCP  demonstration  site. 


Same  surface  model  as  at  left  with  superimposed 
lidar  points,  showing  that  the  “corduroy”  lines  are 
congruent  with  the  lines  of  lidar  points. 


Orthophoto  and  lidar  surface,  municipal  airport,  Washington  State.  The  lidar  surface  shows  more  “texture"  than 
the  flat  surfaces  of  the  parking  lot  and  building  roofs. 


Small  precision  errors  (under  around  5  cm),  arise  from  small  errors  in  the  sensor  system 
and  represent  the  inherent  limits  of  the  technology.  Larger  precision  errors,  including 
corduroy  striping,  also  may  be  caused  by  other  factors,  including  calibration  errors, 
oversampling  the  data,  moderate  to  severe  turbulence,  asymmetrical  spot  spacing, 
excessive  noise  in  the  sensor  apparatus,  and  GPS  error.  According  to  some  vendors, 
the  most  common  cause  of  larger  precision  errors  such  as  corduroy  striping  is  calibration 
error.  Calibration  errors  are  caused  by  a  systematic  “drift”  in  the  IMU  (roll,  pitch,  crab), 
which  remains  fairly  uniform  over  the  course  of  a  day.  Calibration  errors  are  corrected 
for  in  post-processing.  Calibration  must  be  performed  after  each  collection  and  is  the 
primary  method  for  increasing  precision  of  lidar  data.  The  process  works  by  assessing 
the  precision,  or  the  internal  agreements  among  successive  passes,  over  a  series  of 
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properly  aligned  symmetrical  targets  and  then  developing  a  set  of  coefficients  to  correct 
for  the  consistent  offset  observed  in  the  data.  Assuming  high  quality  GPS  data,  when 
calibration  is  performed  properly,  the  range  in  elevation  values  over  (for  example)  a 
perfectly  flat  surface  will  be  the  same  for  one  flight  line  as  it  is  for  multiple  passes  over 
the  same  target.  If  this  internal  agreement  is  not  consistent  then  a  calibration  error 
should  be  suspected.  Because  calibration  must  be  performed  on  each  data  set  and 
because  the  effects  of  a  poor  calibration  are  always  substantial,  it  must  be  assumed  that 
calibration  is  among  the  top  contributors  to  poor  precision  and  therefore  the  cause  of  the 
corduroy  striping7. 

Assessing  the  precision  of  lidar  points.  Lidar  system  manufacturers  generally  report 
overall  accuracies  for  the  system  as  a  whole.  No  results  have  been  located  for  precision 
measurements  alone,  such  as  bench  tests  of  ranging  error  under  fully  controlled 
conditions,  and  in  practice  isolating  the  contribution  of  all  of  the  factors  contributing  to 
precision  error  would  be  difficult  to  achieve.  However,  precision  of  lidar  under  field 
conditions  can  be  roughly  assessed  by  examining  the  elevation  values  of  adjacent  lidar 
points  or  rows  of  points  on  reasonably  flat  surfaces  such  as  parking  lots,  roads,  or 
building  roofs. 

URS  conducted  an  evaluation  using  a  set  of  12  lines  of  lidar  points  at  the  Former  Camp 
Beale  ESTCP  demonstration  site  (Figure  4).  The  site  chosen  was  a  road  surface, 
appearing  from  the  orthophoto  to  be  in  good  condition,  and  located  parallel  to  the  flight 
lines  and  thus  perpendicular  to  the  lines  of  laser  returns.  The  assumption  of  this  small 
test  is  that  by  using  a  small  lidar  data  set  with  points  located  close  together  on  a 
reasonably  flat  surface;  the  influence  of  terrain  changes  along  the  road  would  be 
minimized,  resulting  in  a  rough  snapshot  of  precision8. 

The  distance  between  the  vertical  lines  of  points  in  Figure  4  was  from  5-19  cm,  with  the 
total  distance  along  the  road  of  approximately  3  m.  Mirror  speed  during  data  acquisition 
was  30  hertz  (Hz),  so  the  12  lines  of  lidar  points  in  the  test  set  were  collected  in 
approximately  0.4  seconds,  and  were  thus  based  on  either  one  or  two  GPS  readings. 


7  Sky  Research,  2009. 

8  The  test  could  be  improved  by  choosing  a  more  truly  flat  surface  at  an  area  with  vertical  scan  angles, 
which  were  not  controlled  in  this  data. 
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Figure  4:  Precision  in  Adjacent  Lidar  Points 
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Lidar  points  on  road  surface,  Former  Camp  Beale  ESTCP  demonstration  site. 


Elevation  differences  along  and  between  adjacent  lidar  points. 
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The  test  showed  the  following  results: 

■  Average  point-to-point  difference  along  the  lines  of  points  of  5.1  cm  (minimum  0, 
maximum  16  cm) 

■  Average  point-to-point  difference  between  the  lines  of  points  of  7.4  cm,  (minimum 
0,  maximum  21  cm),  measured  as  the  elevation  difference  between  the  lidar 
point  and  the  nearest  lidar  point  on  the  adjacent  line 

Some  of  this  discrepancy  between  lidar  point  elevations  is  likely  due  to  unevenness  of 
the  road  surface  itself;  however,  this  test  appears  to  indicate  the  approximate  level  of 
point-to-point  error.  These  results  are  consistent  with  informal  examination  of  “corduroy” 
striping  in  flat  desert  surfaces  of  the  ESTCP  demonstration  sites,  where  lines  of  lidar 
points  causing  the  “corduroy  stripes”  appear  to  have  an  elevation  difference  of 
approximately  5-8  cm. 

2.5  Sources  of  Lidar  Point  Error 

Several  factors  influence  the  accuracy  and  precision  of  individual  lidar  points.  In  most 
cases,  the  magnitude  of  each  error  source  is  not  well  documented,  since  vendors  and 
manufacturers  generally  report  error  values  for  the  system  as  a  whole. 

2.5.1  Instrument  Error 

Broadly,  instrument  error  refers  to  the  difference  between  the  value  given  by  an 
instrument  and  the  “actual”  values,  based  on  the  accuracy  and  precision  of  the 
measuring  instrument.  In  the  context  of  lidar,  instrument  error  consists  of  deviations 
from  positional  values,  generally  as  determined  by  GPS-based  survey  methods, 
contributed  by  each  physical  component  of  the  system.  Instrument  error  can  impact 
both  accuracy  and  precision. 

System  components.  In  order  to  establish  the  location  of  the  lidar  points,  the  lidar 
sensor  system  must  establish  the  location  of  the  sensor  in  space  and  the  distance  and 
angle  from  the  sensor  to  the  point  of  reflection  of  the  laser  pulse. 

The  position  of  the  aircraft  is  determined  using  the  GPS  and  Inertial  Navigation  System 
(INS).  The  GPS  samples  the  aircraft  location  based  on  the  position  of  at  least  four 
orbiting  navigation  satellites,  at  rates  between  one  and  ten  times  per  second.  Vendors 
report  that  error  in  the  location  of  the  aircraft  is  primarily  GPS  error,  with  the  most 
serious  source  of  GPS  error  being  the  momentary  loss  of  sufficient  satellite  data  to 
establish  locations  accurately. 

The  INS  calculates  the  position  of  the  aircraft  between  GPS  locations.  The  primary  input 
to  the  INS  is  from  the  Inertial  Measurement  Unit  (IMU).  ).  The  IMU  works  by  detecting 
the  changes  to  rates  of  acceleration,  along  with  rotational  attributes  such  as  pitch,  roll, 
and  yaw.  The  IMU  generally  contains  three  accelerometers  and  three  gyroscopes,  each 
placed  in  orthogonal  positions  so  that  data  is  collected  in  all  three  planes.  Current  IMUs 
record  these  changes  at  200  hertz  (Hz)  (200  samples  per  second).  The  IMU  adds  a 
small  error  component  in  each  plane.  This  error  results  from  the  accumulation  of  small 
errors  as  positions  are  continually  re-calculated.  These  small  errors  accumulate  until  the 
aircraft’s  position  is  updated  from  the  GPS. 
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The  laser  system  consists  of  a  laser,  a  receiver,  and  a  mirror  that  directs  each  pulse 
towards  the  ground  surface.  Lidar  systems  typically  use  lasers  with  wavelengths  of 
1 ,000  to  1 ,500  nanometers.  The  mirror  system  may  be  rotating  or  oscillating  depending 
on  the  manufacturer.  The  receiver  is  a  passive  device  that  is  tuned  to  the  frequency  of 
the  laser,  which  records  a  signal  when  the  amplitude  of  light  in  that  frequency  exceeds  a 
threshold  value.  The  sensor  also  records  the  energy  level  of  the  return,  which  is  referred 
to  as  its  intensity  value.  The  threshold  energy  value  that  will  trigger  a  return  may  or  may 
not  be  adjustable,  depending  on  the  manufacturer. 

The  receiver  can  record  multiple  returns  from  each  laser  pulse.  In  vegetated  conditions, 
part  of  the  laser  return  may  be  reflected  from  branches  and  other  sub-canopy  features, 
leaving  the  remainder  of  the  signal  to  be  reflected  from  the  ground  surface.  The 
limitation  of  the  multi-return  capability  is  that  there  must  be  sufficient  time  between 
returns  for  the  sensor  to  reset.  This  time  is  generally  between  3  and  20  nanoseconds 
and  depending  on  the  sensor  used,  a  distance  of  between  0.5  and  3  m9.  Thus,  multiple 
returns  cannot  be  recorded  in  low  vegetation. 

The  final  components  of  the  sensor  system  are  the  power  source,  the  hardware-  and 
software-based  control  system,  and  the  data  storage  equipment  (consisting  of  multiple 
high-speed  hard  drives). 

The  lidar  system  has  some  parameters  that  can  be  varied,  all  within  the  equipment 
specifications  of  each  manufacturer.  These  include  power  level,  laser  pulse  rate,  mirror 
speed,  and  scan  angle  or  field  of  view.  Vendors  report  that  the  primary  parameters  that 
are  varied  are  pulse  rate  and  scan  angle. 

A  relatively  recent  development  in  lidar  technology  is  the  analog  or  “full  wave  form”  laser 
receiver,  available  from  several  manufacturers.  In  contrast  to  the  approach  described 
above,  the  analog  receiver  records  a  continuous  level  of  energy  values  once  the 
amplitude  passes  the  threshold  value.  The  shape  of  this  energy  return  curve  can  be 
analyzed  and  multiple  return  values  derived  in  a  more  interactive  manner.  Analog 
receivers  create  much  larger  data  sets  than  traditional  sensors,  and  more  advanced 
software  is  needed  to  process  the  output.  Analog  sensors  are  typically  used  for 
analyzing  vegetation  rather  than  modeling  the  ground  surface,  and  appear  to  offer  little 
advantage  at  most  munitions  sites.  Since  analog  receivers  are  not  commonly  used  at 
sites  similar  to  military  munitions  sites,  this  paper  does  not  describe  them  in  further 
detail.  However,  error  in  these  systems  appears  to  derive  from  similar  sources  and  to 
have  similar  magnitudes  as  the  systems  described  here. 

Overall  instrument  package  error.  Manufacturers  and  vendors  typically  report  error 
for  the  entire  system,  expressed  as  the  difference  between  the  positions  of  individual 
lidar  points  compared  to  surveyed  positions.  Reported  error  for  two  example  systems  is 
presented  below.  Both  systems  are  in  common  use  in  the  industry  and  represent  typical 
error  values.  This  section  assumes  that  the  instruments  are  calibrated  and  used 
appropriately  and  are  functioning  correctly. 

■  Optech  Canada  ALTM  3100EA.  Optech  is  one  of  the  largest  builders  of  lidar 
systems  at  the  time  of  this  report.  The  Optech  product  reviewed  is  a  complete 
instrument  package.  Its  published  accuracy  specifications  assume  a  50°  field  of 


9  Terra  Remote  Sensing  personal  communication,  2009 
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view  in  standard  atmospheric  conditions.  Operational  altitudes  are  between  80 
to  3,500  m  above  ground  level.  Although  not  shown  in  Figure  5,  vertical 
accuracy  are  stated  to  be  between  5  and  20  cm. 

Horizontal  accuracy  varies  with  altitude;  the  formula  for  horizontal  error  is: 
1/5,500  x  altitude  (m  above  ground  level).  Both  horizontal  and  vertical  error  is 
given  at  one  standard  deviation.  Horizontal  accuracy  values  vary  from  under  10 
cm  to  over  60  cm  depending  on  flight  height,  as  illustrated  in  Figure  5. 

Figure  5:  Optech  ALTM  3100EA  Horizontal  Error  Curve 


Flight  Height  (m) 


■  Leica  Geosystems  ALS60  Airborne  Laser  Scanner.  Leica  is  another  major 
builder  of  lidar  systems  and  the  Leica  ALS60  is  also  a  complete  instrument 
package.  Leica  published  the  accuracy  specifications  shown  in  Figure  6;  their 
estimates  are  based  on  a  40-degree  field  of  view  and  a  nominal  5  cm  GPS  error. 
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Figure  6:  Leica  ALS60  Horizontal  and  Vertical  Error  Curve 


0  1000  2000  3000  4000  5000  6000 

Flight  Height  (m) 

The  Leica  chart  is  more  complete  that  the  Optech  chart,  and  shows  that  both 
horizontal  and  vertical  error  increase  not  only  with  flight  height,  but  with  distance 
from  nadir.  The  chart  shows  that  vertical  (z)  error  ranges  from  under  10  cm  to 
over  20  cm  depending  on  flight  height  and  distance  from  nadir,  and  that 
horizontal  (x,y)  error  ranges  from  around  5  cm  to  over  60  cm,  again  depending 
on  flight  height  and  distance  off  nadir. 

The  Leica  chart  shown  above  is  not  in  accord  with  the  observation  by  Hodgson 
and  Bresnahan  (2004),  which  states  that  “horizontal  error  is  often  reported  to  be 
approximately  1/1, 000th  of  the  flying  height  AGL  [above  ground  level]  on  most 
systems.”  Rather,  the  Leica  chart  shows  horizontal  error  at  1,000  m  AGL  of 
approximately  1/1 0,000th  of  the  flight  height. 

These  error  relationships  are  assumed  to  be  roughly  similar  for  all  lidar  systems,  based 
on  the  similarity  in  the  technology  used  and  the  relative  consistency  in  the  accuracies 
guaranteed  by  vendors  in  their  contract  documents.  .Within  the  overall  instrument  error 
charts  presented  above  are  a  variety  of  subcategories  of  error  caused  by  individual 
system  components.  Manufacturers  typically  do  not  specify  the  error  contribution  of 
each  component  (Glennie  2007),  however,  types  of  error  are  further  described  in  Habib 
and  Van  Rems  (2009). 

Site  Conditions.  Error  from  site  conditions  refers  to  the  impact  of  local  site  conditions 
on  the  accuracy  and  precision  of  the  lidar  points,  and  thus  on  all  of  the  resulting 
products.  A  wide  variety  of  local  conditions  can  create  error  in  lidar  data.  Some 
conditions  impact  the  entire  data  set,  either  by  affecting  the  equipment  directly  or 
interfering  with  the  GPS  signal.  Others,  such  as  terrain  roughness,  may  impact  only  part 
of  the  survey  area.  Site  conditions  that  can  contribute  to  lidar  error  include: 
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Slope.  Lidar  collected  on  steep  terrain  will  be  less  accurate  than  that  on  flat  terrain 
(Figure  7). 


Figure  7:  Effects  of  Slope  on  Lidar  Point  Error 


■  The  lidar  pulse  spreads  as  the  beam  travels.  The  degree  of  beam  divergence  is 
affected  by  the  characteristics  of  the  laser  combined  with  the  flight  height,  with 
typical  beam  footprints  between  10  and  100  cm  in  diameter10. 

Beam  divergence  creates  an  area  of  uncertainty  as  to  the  exact  point  where 
enough  energy  is  reflected  to  trigger  a  return.  On  flat  surfaces  this  uncertainty 
may  affect  the  horizontal  location  of  the  return,  but  will  not  affect  the  elevation 


10 

Formula  for  beam  divergence  is  roughly:  spot  diameter  at  nadir  =  elevation  (meters)*beam  divergence  (radians) 
(Baltsavias  1999).  System  manufactures  report  beam  divergence  factors  from  0.22  to  1 .0  mrad  (Key  2009).  Laser 
footprints  would  therefore  range  from  22  to  100  cm  at  1,000  m  flight  elevations. 
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value.  On  sloping  surfaces,  the  area  of  the  pulse  footprint  will  be  larger,  and  will 
also  include  a  vertical  error  range. 

■  The  horizontal  error  of  the  lidar  location,  caused  by  both  beam  divergence  and 
the  error  inherent  in  the  GPS,  IMU  and  other  components,  will  be  larger  on 
sloped  ground.  Further,  on  sloped  ground  the  horizontal  error  will  magnify  the 
vertical  error  versus  flat  ground.  The  maximum  amount  of  elevation  error 
introduced  is  a  function  of  surface  slope,  with: 

Elevation  Error  =  tan  a  x  Horizontal  Displacement 

Studies  have  shown  that  slope  errors  in  lidar  horizontal  and  vertical  locations  increased 
consistently  with  increasing  slope  (Hodgson  et  al  2005,  Bowen  and  Waltermire  2002, 
Maling  1989). 

Interaction  of  slope  and  scan  angle.  At  the  edges  of  the  scan,  the  laser  signal  will 
approach  the  ground  at  an  angle.  This  can  either  multiply  or  cancel  the  effect  of  the 
ground  slope.  Some  experimentation  has  been  done  with  estimating  the  combined 
impact  of  terrain,  incidence  angle  and  beam  width  (Schaer,  et.al.  2007). 

Ground  surface.  Some  vendors  guarantee  higher  accuracy  on  hard  surfaces  than  on 
soft  ground  surfaces;  this  is  based  on  the  fact  that  soft  surfaces  are  often  less  clearly- 
definable.  Plowed  fields  or  grassy  areas,  for  instance,  would  have  furrows  or  vegetation 
that  would  create  error  up  to  10-20  cm  (Terrapoint,  2009). 

Similarly,  it  is  occasionally  observed  that  highly  reflective  surfaces  will  appear  to  be 
slightly  raised  in  comparison  to  non-reflective  surfaces,  such  as  white  painted 
centerlines  on  asphalt  roadways.  It  is  possible  that  more  reflective  surfaces  are 
returning  sufficient  energy  to  trigger  a  response  more  quickly  than  less  reflective 
surfaces. 

Vegetation  conditions.  Under  vegetated  conditions,  it  is  common  for  the  majority  of 
laser  signals  to  reflect  from  the  vegetation  rather  than  the  ground  surface  (Figure  8),  the 
lower  density  of  lidar  points  under  vegetation  will  lead  to  lower  confidence  levels  for  the 
accuracy  of  the  lidar  points,  since  the  elevation  of  the  laser  reflection  will  be  compared  to 
a  more  coarse  surface  model. 

Electromagnetic  interference.  Rarely,  certain  types  of  electromagnetic  signals  can 
affect  lidar  data  collection.  This  can  include  the  on-board  radio  system,  which  can 
interfere  with  GPS  antenna  performance,  and  external  sources.  External  interference 
was  noted  at  the  Former  Camp  Beale  ESTCP  demonstration  site.  Beale  Air  Force  Base, 
which  is  adjacent  to  the  demonstration  site,  is  the  site  of  one  of  three  installations  that 
are  part  of  the  Phased  Array  Warning  System  (PAWS),  a  radar  system  designed  to 
detect  and  track  sea-launched  ballistic  missiles.  The  high-intensity  radar  signals  from 
this  installation  disrupted  the  GPS  time  signal  used  by  the  lidar  system,  initially  rendering 
the  data  unusable.  The  effect  was  noted  within  several  kilometers  of  the  radar  station 
and  at  altitudes  up  to  approximately  500  m.  The  problem  was  noted  during  the  daily 
quality  assurance/quality  control  checks  on  the  first  day  of  data  collection.  A  sample  of 
the  data  was  sent  to  the  vendor’s  office,  and  a  solution  was  developed  to  re-insert  the 
correct  GPS  times.  Interestingly,  lidar  was  subsequently  collected  at  Beale  Air  Force 
Base  without  problems,  since  lidar  data  collection  took  place  at  a  higher  altitude  outside 
of  the  influence  of  the  radar  system. 
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Figure  8:  Percentages  of  Laser  Points  Reflecting  from  Vegetation 
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Percentage  of  lidar  points  reflected  from  vegetation,  darker  green  and  blue  areas  have  higher  percentages  of  vegetation 
returns.  Although  ground  detail  is  visible  under  the  tree  cover,  the  lower  number  of  ground  returns  will  result  in  lower 
confidence  in  these  areas. 


Weather  and  temperature.  Weather  and  temperature  conditions  can  affect  the  ability 
to  collect  lidar  data,  but  should  not  affect  the  accuracy  or  precision  of  the  lidar  data  itself. 

2.5.2  Operator  Error 

Potential  areas  where  actions  of  the  operator  can  impact  lidar  products  are  potentially 
numerous  and  can  include  the  following  (Sky  Research,  2009): 

■  Improper  planning  that  can  cause  an  asymmetric  distribution  of  points  on  the 
ground 

■  Failure  to  inform  the  pilots  when  they  are  offline,  which  can  result  in  holes  in  the 
data 

■  Collection  during  periods  of  elevated  position  dilution  of  precision  (PDOP)  that 
exceed  contract  specifications,  which  may  result  in  noisy  or  inferior  GPS  data 

■  Exceeding  roll  restrictions  that  can  manifest  in  a  poor  position  and  orientation 
solution 

There  are  many  sources  for  operator  error,  including  inexperience  or  improper  training, 
momentary  lack  of  judgment,  inattentiveness,  or  distraction.  Operator  error  also  may 
result  from  incomplete  information,  as  when  the  operator  does  not  fully  understand  the 
purpose  of  the  lidar  survey.  Operator  error  is  difficult,  if  not  impossible  to  quantify. 

The  standard  approach  to  eliminating  operator  error  is  through  the  application  of  training 
programs  to  reduce  error  and  quality  control  programs  to  detect  and  correct  errors.  Most 
vendors  have  well  developed  training  and  quality  assurance  programs,  and  the  impact  of 
serious  operator  error  is  generally  small.  This  is  particularly  true  in  regard  to  all 
processes  that  impact  the  positional  accuracy  of  the  lidar  data,  where  operator  error  may 
cause  the  vendor  to  fail  to  meet  contract  specifications. 
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2.6  Implications  for  Munitions  Investigations 

Instrument  Error.  The  charts  presented  above  show  that  the  accuracy  of  lidar  points 
varies  primarily  with  flight  height  and  distance  from  nadir.  Lidar  acquisitions  at  the 
ESTCP  demonstration  sites  were  conducted  from  300  to  1,000  m,  where  instrument 
error  is  relatively  low.  Lidar  collected  at  these  sites  met  their  contracted  accuracy 
specifications. 

At  the  ESTCP  demonstration  sites,  features  with  elevation  differences  less  than  10  cm 
were  successfully  detected,  including  bombing  targets  and  potential  craters  (URS  2007). 
These  features  were  detected  despite  the  fact  that  the  vertical  accuracy  specification 
was  15  cm.  This  is  most  likely  due  to  the  fact  that  vertical  precision  error  is  generally 
lower  than  accuracy  error. 

Subsequent  acquisitions  in  this  range  of  flight  height  should  successfully  meet  similar 
accuracy  specifications.  However,  where  accuracy  is  especially  important,  contract 
specifications  may  be  adjusted  to  require  the  vendor  to  fly  lower  and/or  use  a  narrower 
field  of  view. 

Site  Conditions.  Slope  is  the  site  factor  with  the  most  direct  impact  on  the  accuracy  of 
the  individual  lidar  points,  and  this  impact  can  be  relatively  severe.  The  effects  of  slope 
and  other  site  conditions  such  as  vegetation  cover  are  generally  documented  by  placing 
additional  calibration  points  in  all  major  terrain  types.  Data  calibration  would  still  be 
performed  using  the  most  accurate  points,  i.e.  those  on  flat,  un-vegetated  surfaces,  with 
remaining  points  used  for  quality  control  and  accuracy  evaluation. 

Operator  Error.  Most  operator  error  can  be  controlled  through  applying  accuracy 
specifications  to  the  lidar  data.  Serious  operator  error  will  result  in  data  that  fails  to  meet 
specifications,  and  which  will  therefore  not  be  accepted.  However,  an  important 
supplement  to  contract  specifications  is  to  be  sure  that  data  collection  and  processing 
staff  (not  just  contracting  staff  or  management)  fully  understand  the  goals  of  the  survey 
and  the  objects  to  be  detected,  so  that  collection  and  processing  methods  can  be 
appropriately  adjusted. 
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3  GPS  Error 


GPS  error  is  a  component  of  the  overall  lidar  system  error  discussed  above.  Error  in 
GPS  is  discussed  separately  for  two  reasons.  First,  vendors  report  that  GPS  error  is  the 
largest  source  of  error  in  the  day-to-day  use  of  lidar  technology.  Second,  GPS  error 
affects  not  only  the  positional  accuracy  of  the  lidar  points,  but  the  accuracy  of  field  data, 
including  both  calibration  points  and  the  reported  locations  of  features  in  the  field. 

3.1  Theory  of  Operation 

The  United  States  GPS  system  is  based  on  a  network  of  32  satellites  operated  by  DoD; 
the  system  has  been  operational  since  1978.  The  satellites  are  stationed  in  six  orbital 
rings,  with  approximately  circular,  non-geostationary  orbits  with  radii  of  26,560 
kilometers  and  orbital  periods  of  approximately  1 1 .9  hours.  A  complementary  system  is 
the  Russian  GLOASS  system,  which  was  planned  for  24  satellites.11  Similar  systems 
are  planned  by  China  (Wikipedia  2009a)  and  the  European  Union  (Wikipedia  2009b). 
Some  GPS  receivers  are  configured  to  receive  signals  from  both  satellite  groups,  and 
accuracy  of  such  dual  systems  is  reported  to  be  higher  (Clarkin  2007)12. 

Each  GPS  satellite  sends  out  two  carrier  waves,  and  each  also  transmits  a  unique 
“Coarse  Acquisition,  pseudo-random  noise”  code,  which  is  stored  in  the  library  of  GPS 
receivers  and  which  modulates  the  first  carrier  wave. 

A  GPS  receiver  must  lock  onto  and  track  at  least  four  satellites  in  order  to  determine  a 
position  on  the  earth.  A  minimum  of  three  receivers  are  needed  to  determine  the  x,  y 
and  z  coordinates  (more  are  desirable)  and  a  fourth  is  required  to  solve  for  clock  error 
between  the  atomic  clocks  aboard  the  satellites  and  the  less  accurate  clocks  built  into 
the  receivers. 

“Differential  correction”  is  a  method  of  reducing  systematic  positional  error  by  applying 
corrections  from  outside  sources.  The  most  common  form  of  differential  correction  is 
achieved  through  the  use  of  a  stationary  GPS  base  station  at  a  surveyed  location.  The 
base  station  monitors  its  location  based  on  the  satellites  visible  at  each  interval,  and 
then  calculates  the  difference  between  this  measured  location  and  its  surveyed  location. 
These  differences  are  then  applied  to  field  locations  recorded  by  the  receiver.  When  this 
differential  correction  is  done  in  real  time  it  is  referred  to  as  real  time  kinematic 
differential  correction,  and  when  the  differences  are  applied  following  data  collection  it  is 
referred  to  as  post-processed  differential  correction.  Accuracies  quoted  here  assume 
the  use  of  differential  correction  except  for  recreation  grade  receivers. 

3.2  GPS  Error  Sources 

Potential  sources  of  GPS  error  have  been  summarized  in  several  sources.  The 
following  discussion  is  taken  from  Clarkin  (2007).  System  specifications  provided  by 


As  of  February  2009,  the  system  was  reported  to  have  20  satellites,  of  which  19  were  operational  and  one  was 
undergoing  maintenance  (Wikipedia  2009c). 

12  A  description  of  the  GLONASS  system  (provided  by  a  receiver  manufacturer)  can  be  found  at: 
http://www.novatel.com/Documents/Papers/GLONASSOverview.pdf 
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manufacturers  include  combined  error  from  all  of  these  sources,  and  little  quantitative 
data  was  located  regarding  the  relative  contribution  of  each  of  these  sources  separately. 
Factors  contributing  to  GPS  error  include: 

■  Ionospheric  delay.  The  ionosphere  is  a  portion  of  the  atmosphere  that  ranges 
from  50-1000  kilometers  (km)  and  contains  gases  that  are  ionized  by  solar 
radiation.  The  production  of  electrons  from  these  gasses,  measured  as  total 
electron  content,  cause  delays  in  the  propagation  of  satellite  signals  causing 
errors  on  the  order  of  10-20  m  (Grewal  et  al.  2001).  Although  dual-frequency 
users  can  easily  resolve  this  problem,  ionospheric  models  still  need  to  be 
developed  for  single-frequency  users  (Le  and  Tiberius  2006).  Dual-frequency 
receivers  nearly  eliminate  ionospheric  effects  by  comparing  the  propagation  of 
the  signal  at  two  frequencies  (Leick  2003).  Because  the  delay  induced  by  the 
ionosphere  is  known  to  be  inversely  proportional  to  the  square  of  frequency, 
ionospheric  range  error  can  be  estimated  accurately  by  comparing  the  times  of 
arrival  of  the  LI  and  L2  signals  (Grewal  et  al.  2001). 

■  Tropospheric  delay.  The  troposphere  is  the  lower  part  of  the  earth’s 
atmosphere  and  also  can  delay  the  propagation  of  Global  Navigation  Satellite 
System  (GNSS)  signals,  due  to  dry  gasses  and  water  vapor  refracting  the 
signals.  The  error  is  far  less  than  that  of  ionospheric  delay  (1-3  m)  and  generally 
can  be  eliminated  by  differential  correction  and  modeling  (Grewal  et  al.  2001). 

■  Multipathing.  Multipathing  refers  to  the  reflection  of  satellite  signals  off  of 
objects  or  surfaces,  producing  signals  of  multiple  ranges  arriving  at  a  GNSS 
receiver.  Reception  of  satellite  signals  from  multiple  pathways,  in  addition  to 
direct  reception  paths,  can  cause  large  coarse  acquisition  code  ranging  errors  of 
up  to  approximately  10  m  and  degrade  the  ambiguity  resolution  process  required 
in  carrier-phase  ranging  (Grewal  et  al.  2001).  Since  multipathing  effects  are 
specific  to  receiver  location,  these  errors  cannot  be  reduced  by  differential 
correction.  Piedallu  and  Gegout  (2005)  found  that  errors  caused  by  multi-path 
signals  due  to  signal  reflections  off  trees  were  much  greater  than  the  errors  that 
differential  corrections  could  reduce.  They  contended  that  in  forest 
environments,  the  multipath  of  the  signal  caused  by  the  tree  stand  is  the  main 
error  source. 

■  Elevation  angle.  Elevation  angle  is  the  angle  between  a  satellite  signal  and  the 
horizon.  It  is  well  known  that  the  signal  received  at  a  low  elevation  angle  will  be 
more  affected  by  multipath,  ionosphere,  and  troposphere  errors  and  receiving 
antenna  gain  patterns  (Parkinson  and  Spilker  1996,  Le  and  Tiberius  2006).  The 
effect  of  low  elevation  angle  can  be  even  more  severe  when  collecting  GNSS 
information  in  heavily  forested  environments.  For  environmental  field  studies,  it 
is  a  rule  of  thumb  that  satellites  lower  than  10-15  degrees  above  the  horizon 
should  not  be  used  for  positioning  because  of  atmospheric  refraction  (Johnson 
and  Barton  2004).  Most  receivers  have  an  elevation  mask  setting  that  can  be 
used  to  specify  the  lowest  elevation  angle  that  will  be  accepted. 

■  Satellite  constellation  and  position  dilution  of  precision.  Position  dilution  of 
precision  (PDOP)  is  a  quality  measure  of  satellite  geometry.  When  the  satellites 
contributing  to  a  GNSS  position  are  spread  out,  they  provide  a  low  PDOP  value 
and  a  strong  geometry  for  a  better  solution.  When  satellites  are  close  together, 
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they  provide  high  PDOP  values  and  weaker  solutions.  Both  theory  and  research 
have  shown  that  accuracy  decreases  as  PDOP  increases  (Karsky  et  al.  2001), 
so  filtering  out  occupations  with  high  PDOP  values  can  be  very  important  when 
taking  GNSS  positions.  This  is  especially  important  in  obstructed  environments. 
Piedallu  and  Gegout  found  that  in  closed  environments  (coppice  and  high  forest) 
the  positioning  errors  increase  linearly  or  according  to  an  exponential  model  with 
the  PDOP  (Piedallu  and  Gegout  2005).  Most  receivers  have  a  PDOP  mask 
setting  that  can  be  used  to  specify  the  highest  PDOP  that  will  be  accepted. 

■  Signal-to-noise  ratio.  Signal-to-noise  ratio  is  the  ratio  of  satellite  signal  strength 
to  the  strength  of  the  surrounding  noise  measured  in  band  width  relative  to  1  Hz 
(dB-Hz).  The  canopy  has  been  shown  to  have  a  substantial  effect  on  total  signal 
loss  and  the  reduction  of  signal-to-noise  ratio  (Karsky  et  al.  2001 ).  Rodriguez- 
Perez  et  al.  (2006)  also  found  that  positional  accuracy  was  affected  by  stand 
density  because  of  the  lowering  of  signal-to-noise  ratio  and  signal  interception 
caused  by  the  electromagnetic  waves  penetrating  the  stem  and  canopies. 
However,  other  research  has  found  that  a  vegetation  canopy  may  produce 
complete  attenuation  to  little  or  no  apparent  effect  on  the  signal  pathway 
(Gerlach  and  Jasumback  1989).  Furthermore,  GNSS  signals  become  unusable 
when  the  ratio  falls  below  approximately  25  dB-Hz  (Grewal  et  al.  2001).  Most 
receivers  have  a  signal-to-noise  ratio  mask  setting  that  can  be  used  to  specify 
the  highest  PDOP  that  will  be  accepted. 

■  Geomagnetic  activity.  Geomagnetic  activity  refers  to  natural  variation  in  the 
earth’s  magnetic  field,  usually  as  a  result  of  solar  flares  and  other  solar  activity. 
During  periods  of  high  geomagnetic  activity  the  GPS  phase  can  be  affected  to 
the  point  where  the  receiver  cannot  perform  phase  measurements  with  enough 
precision  for  centimeter  level  accuracy.  It  is  good  practice  to  check  government 
web  sites  for  geomagnetic  activities  prior  to  data  collection13. 

■  Occupation  length  and  logging  rate.  Occupation  length  refers  to  the  amount 
of  time  spent  recording  data  at  a  GNSS  point  (and  consequently  the  number  of 
observations  that  will  be  averaged  to  determine  the  final  position.  Occupation 
lengths  in  GNSS  research  range  from  1  minute  up  to  24  hours  (Sawaguchi  et  al. 
2003).  Logging  rate,  which  is  the  frequency  of  GNSS  positions  recorded,  is 
closely  associated  with  occupation  length.  For  instance,  Piedallu  and  Gegout 
2005)  found  that  positioning  errors  decrease  linearly  with  the  logarithm  of  the 
number  of  points  taken  into  account  when  carrying  out  the  measurement. 
Sawaguchi  et  al.  (2003)  found  that  over  1,000  logged  positions/sample  numbers 
(~17  minutes)  were  needed  to  achieve  a  precision  to  within  1  m.  Comparatively, 
Naesset  (2001 )  found  in  one  study  that  the  accuracy  of  GNSS  point  positions  did 
not  seem  to  improve  beyond  15  minutes  of  observation.  The  National  Geodetic 
Survey  also  states  that  experiments  show  that  multipathing  can  take  10  minutes 
or  more  to  average  out  in  recording  an  observation.  However,  Hasegawa  and 
Yoshimura  (2003)  found  that  the  probability  of  resolving  ambiguities  in  carrier- 
phase  solutions  significantly  improved  up  to  an  observation  period  of  30  minutes 
or  slightly  more. 


13  See,  for  instance,  the  NOAA  Space  Weather  Prediction  Center  at 
http://www.swpc.noaa.gov/forecast.html. 
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GPS  field  practices  are  designed  to  minimize  these  error  sources. 

3.3  Receiver  Categories 

There  are  three  broad  categories  of  GPS  receivers  on  the  market.  The  first  is  recreation 
or  consumer  grade,  available  from  manufacturers  such  as  Magellan  and  Garmin.  These 
receivers  generally  cost  under  $1,000  and  use  the  coarse-acquisition  code  to  obtain 
rough  positions.  Positional  accuracies  are  quoted  as  3-5  m  in  open  areas14. 
Manufacturers  do  not  quote  accuracies  for  obstructed  areas  such  as  under  vegetation 
canopy  for  any  type  of  GPS  receiver  (Clarkin  2007);  however,  Wing  and  Eklund  (2007) 
found  these  recreation-grade  receivers  to  be  capable  of  accuracies  within  10  m  under 
closed  canopies  and  7  m  under  young  forest  in  western  Oregon. 

The  second  grade  of  receiver  is  generally  referred  to  as  resource  or  mapping  grade 
receiver;  examples  include  the  Trimble  Pathfinder  ProXTR  or  ProXH.  These  receivers 
generally  cost  under  $5,000  and  usually  receive  single-frequency  carrier  wave  signals  in 
addition  to  the  coarse-acquisition  code.  Accuracy  in  the  open  is  quoted  as  sub-meter 
down  to  around  30  cm15.  Researchers  have  found  single-frequency  receivers  to  achieve 
error  between  0.2  and  6  m  under  vegetation,  depending  on  conditions  (Clarkin  2007). 

The  third  grade  of  receiver  is  referred  to  as  survey  grade;  these  receivers  cost  $20,000 
or  more  and  receive  both  carrier  wave  frequencies,  achieving  sub-centimeter  accuracy 
in  open  conditions.  Hasegawa  and  Yoshimura  (2003)  found  that  survey  grade  receivers 
achieve  accuracies  of  0.02-0.4  m  under  dense  canopy.  Naesset  (2001)  found  a  survey- 
grade  dual  frequency  receiver  had  mean  positional  accuracies  of  0.08-1.35  m  under 
dense  canopy. 

3.4  Implications  for  Munitions  Investigations 

Implications  of  GPS  error  for  munitions  investigations  fall  into  three  categories: 
implications  for  lidar  points,  for  calibration  and  quality  control,  and  for  field  investigations. 

3.4.1  Implications  for  Lidar  Points 

The  published  accuracy  specifications  for  lidar  systems  include  assumptions  regarding 
the  contribution  of  GPS  error.  (Some  are  explicitly  stated,  such  as  the  5  cm  quoted  for 
the  Optech  system  in  Section  2.5.1 ).  As  long  as  GPS  error  does  not  exceed  assumed 
values  quoted  by  vendors,  the  resulting  lidar  data  should  be  within  contracted  accuracy 
specifications.  Monitoring  positional  accuracy  levels  should  be  part  of  the  vendor’s  daily 
quality  assurance/quality  control  procedures. 

3.4.2  Implications  for  Calibration  and  Quality  Control 


For  example,  see  http://www.magellangps.com/products/product.asp?seglD=425&prodlD=1916  for  the  Magellan  Triton 
2000  series,  a  high-end  hand-held  model. 

15  For  example,  see  http://www.trimble.com/pathfinderproxt.shtml  or 

http://store.elecdata.com/trimble/pathfinder_proxh_receiver.aspx.  For  example,  the  Trimble  R8  GNSS,  a  commonly-used 
GPS  system  in  both  static  GPS  and  real  time  kinematic  surveying,  has  published  accuracy  specifications  of  ±5  mm  +0.5 
ppm  horizontally  and  ±5  mm  +1  ppm  vertically  for  static  GPS  applications,  and  ±1  cm  +1  ppm  horizontally  and  ±2  cm  +1 
ppm  vertically  for  real  time  kinematic  applications.  See  Trimble  R8  GNSS  System  Datasheet, 
http://trl.trimble.com/docushare/dsweb/Get/Document-140079/022543-079H_TrimbleR8GNSS_DS_0309_LR.pdf. 


22 


ESTCP  White  Paper:  Errors  in  Lidar  Data:  Implications  for 
Investigation  of  Military  Munitions  Sites 
Project  Number  07  E-MM2-012/MM-0737 


Surveyed  points  used  for  calibration  and  quality  control  must  be  from  a  source  of  higher 
quality  than  the  lidar  data  itself.  Errors  in  the  calibration  points  can  lead  to  incorrect 
assessment  of  data  quality  or  errors  in  the  entire  lidar  data  set.  Standard  practice  in 
establishing  calibration  points  includes: 

■  Using  survey-grade  GPS  operated  by  qualified  staff 

■  Choosing  flat  sites  to  minimize  the  effects  of  slope  between  surrounding  control 
point  and  nearest  lidar  points 

■  Choosing  open  sky  sites,  which  are  essential  to  GPS  quality 

■  Choosing  calibration  points  in  all  major  terrain  types 

■  Reporting  estimated  error  of  all  calibration  and  control  points 

The  only  one  of  these  that  is  not  standard  practice  is  the  placement  of  calibration  points 
in  all  major  terrain  types,  where  in  practice  the  number  of  calibration  points  established 
will  vary  with  the  type  of  survey.  Since  the  accuracy  of  lidar  points  can  be  affected  by 
terrain  (see  Section  2.5),  investigations  at  munitions  sites  should  specify  that  calibration 
points  will  be  established  in  each  major  terrain  type. 

Implications  for  field  Investigations.  Munitions-related  features  such  as  potential 
craters  are  generally  at  least  2  m  in  diameter.  In  most  field  work,  resource-grade  GPS, 
with  its  sub-meter  accuracy,  should  be  able  to  locate  these  features  with  sufficient 
accuracy  to  compare  the  with  lidar  data,  as  long  as  open-sky  conditions  are  present. 

At  vegetated  sites,  however,  the  locations  of  features  collected  by  field  crews  must  be 
treated  with  more  caution,  since  GPS  accuracy  can  be  degraded  under  vegetation.  As 
shown  in  Figure  9,  resource-grade  GPS  can  sometimes  suffer  significant  inaccuracies  at 
vegetated  sites. 


Figure  9:  Potential  Error  in  Resource-Grade  GPS  in  Vegetated  Conditions 


GPS  point  acquired  for  tree  height  investigation,  Former  Camp  Beale  ESTCP  demonstration  site,  orthophoto  left  and 
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lidar  digital  surface  model  right.  The  GPS  point  is  approximately  6  m  from  the  highest  lidar  point  in  the  tree  being 
measured.  GPS  locations  were  acquired  using  resource-grade  GPS.  While  most  GPS  points  acquired  at  this  site 
were  more  accurate  than  the  point  shown,  several  of  the  51  points  acquired  showed  errors  of  this  magnitude. 


For  field  investigations  in  vegetated  conditions,  other  methods,  beyond  the  scope  of  this 
paper,  may  need  to  be  used. 
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4  Digital  Elevation  Models  and  Digital  Surface  Models 


In  practice,  much  of  the  analysis  of  lidar  data  is  performed  using  surface  models  derived 
from  the  lidar  points.  Error  in  the  use  of  lidar  data  should  therefore  include  a  discussion 
of  potential  error  arising  from  the  creation  of  surface  models.  This  type  of  error  may  be 
quantified  by  comparing  the  elevation  of  the  model  to  the  elevation  of  the  lidar  points 
used  to  generate  the  model. 

4.1  Using  Lidar  Points  to  Create  Digital  Surface  Models 

The  typical  final  products  of  lidar  are  digital  models  of  the  ground  surface,  both  of  the 
bare  ground  and  the  ground  with  vegetation  and  buildings  included.  Typical  model  types 
are  digital  elevation  models  (DEMs),  digital  surface  models  (DSMs)  and  digital  terrain 
models  (DTMs). 

A  DEM,  as  defined  by  the  US  Geological  Survey  (USGS),  is  a  digital  file  consisting  of 
terrain  elevations  for  ground  positions  at  regularly  spaced  horizontal  intervals  that 
portrays  the  ground  surface  free  of  vegetation  or  human-created  structures16.  DEMs 
may  be  created  using  data  from  many  sources  in  addition  to  lidar  including  topographic 
maps,  ground  survey,  photogrammetry,  or  synthetic  aperture  radar.  In  the  context  of 
lidar,  the  DEM  is  the  product  through  which  the  semi-random  mass  lidar  points  that  the 
vendor  classifies  as  returning  from  the  ground  surface  are  converted  to  a  regularly- 
spaced  grid  of  elevation  values.  This  digital  model  can  then  be  used  in  standard 
Geographic  Information  System  (GIS)  or  other  software  to  produce  hillshaded  surfaces, 
contour  lines,  or  other  digital  products. 

Regularly  spaced  elevation  files  of  this  type  also  may  be  created  using  all  returns, 
including  those  from  the  tops  of  buildings,  trees  and  other  features.  The  USGS  refers  to 
these  as  DSMs.  The  term  DTM  is  used  as  a  synonym  for  both  DEM  and  DSM.  In  the 
lidar  context,  the  term  DTM  is  most  frequently  used  to  refer  to  the  all-points  surface 
model.  This  paper  uses  the  term  DSM  for  the  all-points  surface  model  created  using 
both  ground  and  non-ground  returns. 

Figure  10  shows  an  example  of  the  lidar  point  cloud,  DSM  and  DEM  for  an  example  site. 

4.2  Error  Sources  in  Digital  Models 

Error  in  the  DEM  can  be  defined  as  the  discrepancy  between  the  elevation  values  of  the 
DEM  cells  compared  to  surveyed  values  at  the  same  locations.  Error  in  the  elevation 
values  of  the  DEM  cells  also  affects  the  horizontal  error  of  features  in  the  ground  surface 
model,  since  it  is  the  elevation  differences  between  adjacent  DEM  cells  that  create 
edges  that  allow  the  user  to  infer  the  location  of  features. 


16  See:  http://rockyweb.cr.usgs.gov/elevation/dpi_dem.html 
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Figure  10:  Digital  Surface  Models  and  Digital  Elevation  Models 


Aerial  orthophoto,  Former  Camp  Beale  ESTCP 
demonstration  site. 


Laser  reflections,  classified  as  ground  (red)  and  non¬ 
ground  (blue).  Modified  vendor  classification. 


Digital  surface  model  using  all  laser  reflections. 


Digital  elevation  model  using  ground  reflections  only. 


Ultimately,  the  accuracy  of  the  DEM  consists  of  its  accurate  depiction  of  the  true  ground 
surface.  Accuracy  of  a  DEM  will  depend  on  the  accuracy  of  the  lidar  data  from  which  it 
is  created,  and  will  be  subject  to  the  types  of  error  that  affect  lidar  points.  However, 
additional  error  can  be  introduced  to  DEMs  through  the  methods  used  to  create  the  DEM 
itself,  including: 


■  The  choice  of  the  points  classified  as  ground  or  non-ground  returns 
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■  The  choice  of  interpolation  method  from  the  lidar  points  to  the  DEM  cell 
elevations 

■  The  choice  of  cell  size 

Each  combination  of  these  choices  may  be  more  accurate  in  a  particular  application. 

4.2.1  Point  Classification  Methods 

The  point  classification  approach  used  by  the  lidar  vendor  should  have  no  impact  on  the 
accuracy  or  precision  of  the  lidar  points,  since  classification  takes  place  after  the  data  is 
calibrated.  However,  methods  that  are  biased  towards  creating  clean,  smooth  ground 
surfaces  can  result  in  a  lower  data  density  of  ground  points,  and  this  may  result  in  fewer 
detections  of  small  ground  features. 

Earlier  work  under  this  contract  demonstrated  that  point  classification  methods  in 
common  use  can  over-classify  lidar  returns  as  non-ground  (USACE  and  URS  2009). 

This  phenomenon  is  a  result  of  classification  routines  that  interact  with  small  elevation 
differences  between  nearby  lidar  points.  The  phenomenon  is  most  common  in  areas  of 
especially  dense  lidar  data.  Including  more  points  in  the  ground  surface  model  allowed 
detection  of  additional  small  surface  features  compared  to  the  vendor’s  original  ground 
surface  model. 

Interviews  indicated  that  vendors  are  capable  of  modifying  point  classification  methods 
increase  the  number  of  points  classified  as  ground,  and  to  adapt  data  collection  and 
calibration  methods  to  minimize  elevation  differences  between  points.  Therefore,  in  the 
context  of  lidar  surveys  of  military  munitions  sites,  the  most  critical  method  for  minimizing 
error  resulting  from  point  classification  methods  is  to  assure  that  vendors  have  a  clear 
understanding  of  the  goals  of  the  lidar  survey,  how  these  differ  from  the  use  of  lidar  for 
other  applications,  and  how  methods  will  be  adapted. 

4.2.2  Digital  Surface  Model  interpolation  Method 

In  a  DSM,  each  cell  of  a  regularly-spaced  grid  is  assigned  an  elevation  value.  This 
elevation  value  is  interpolated  from  the  semi-random  lidar  points  in  and  near  the  grid 
cell.  There  are  two  general  approaches  to  this  process.  The  first  approach  is  to 
interpolate  the  surface  model  directly  from  the  lidar  points;  the  second  is  to  create  a 
Triangulated  Irregular  Network  (TIN)  using  every  individual  lidar  point,  and  to  then 
interpolate  the  surface  model  from  the  TIN. 

Interpolation  is  performed  using  one  of  a  variety  of  mathematical  approaches. 
Interpolation  methods  are  classified  as: 

■  Deterministic  methods  such  as  inverse  distance  weighting,  which  assume  that 
each  input  point  has  a  local  influence  that  diminishes  with  distance 

■  Spline-based  methods  that  fit  a  minimum-curvature  surface  through  the  sample 
points 

■  Geostatistical  methods  such  as  kriging  that  take  into  account  both  the  distance 
and  the  statistical  relationship  among  the  sample  points 

Surveys  of  DEM  interpolation  methods  have  concluded  that  none  of  the  interpolation 
methods  is  universally  superior  for  all  kinds  of  data  sources,  terrain  patterns  or 
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purposes.  Inverse  distance  weighting  is  generally  thought  to  be  suitable  for  lidar  data, 
since  it  works  well  for  dense  and  evenly-distributed  points  (Liu  2008). 

Little  investigation  has  been  done  on  the  effects  of  different  interpolation  methods  at 
munitions  sites  or  for  detection  of  small  surface  features.  Earlier  work  by  URS  examined 
DEM  creation  methods  using  the  data  from  the  Kirtland  Air  Force  Base  Precision 
Bombing  Range  ESTCP  demonstration  site  (URS  2007).  The  points  in  the  area 
surrounding  1 .5  and  1 .0  m  diameter  test  craters  were  extracted  and  used  to  create  a 
series  of  surfaces  using  four  methods.  Deterministic,  inverse  distance  weighted 
interpretation  from  the  TIN  showed  the  calibration  craters  somewhat  more  clearly  than 
surfaces  directly  interpolated  using  other  methods;  however,  the  differences  between  the 
methods  were  not  large,  especially  given  the  small  size  of  the  craters  examined. 

4.2.3  Cell  Size 

Choice  of  cell  size  can  affect  the  detection  of  small  surface  features  dramatically  (Figure 
11). 

Figure  11:  Effects  of  Digital  Surface  Model  Cell  Size  on  Feature  Detection 
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In  Figure  1 1 ,  images  created  with  smaller  cell  sizes  showed  the  potential  craters  with  the 
best  resolution.  Visibility  of  these  features  decreased  with  increasing  cell  size,  and  at  a 
cell  size  of  2  meters  the  features  were  essentially  undetectable. 

It  is  not  always  appropriate  to  use  the  smallest  possible  cell  size.  State-wide  or  region¬ 
wide  lidar  data  sets  typically  have  point  density  of  1-2  pts/m2  and  DEM  cell  sizes  of  1-2 
m17.  The  low  data  density  and  coarse  surface  representation  of  these  surveys  obscure 
much  fine  ground  surface  detail  and  would  not  be  appropriate  for  munitions  sites. 
Nevertheless,  the  resulting  data  is  completely  acceptable  for  its  intended  purpose  of 
mapping  surficial  geology  or  large-scale  stormwater  runoff  or  flooding  patterns.  In  fact, 
in  modeling  surface  water  runoff,  a  highly  detailed  ground  surface  model  often  will  result 
in  incorrect  flow  patterns  when  used  in  standard  models,  and  the  surface  must  be 
artificially  smoothed  or  modeled  with  a  larger  cell  size  (Zandbergen  2006) 

However,  in  investigations  of  munitions  sites,  the  goal  of  the  survey  is  to  detect  small 
surface  features.  In  this  application,  surface  models  should  be  created  using  the 
smallest  cell  size  that  can  be  justified  by  the  density  of  the  lidar  data.  Vendors  contacted 
uniformly  recommended  that  cell  size  be  kept  as  small  as  justified  by  average  point 
spacing. 


4.2.4  Display  Methods 


DEMs  and  DTMs  are  usually  displayed  as  hillshaded  images  to  enhance  the 
visualization  of  surface  features  (Figure  12).  ArcGIS,  the  most  commonly  used  GIS 
software,  contains  dozens  of  settings  that  can  affect  the  usefulness  of  these  images. 
Different  hillshade  settings  may  be  appropriate  in  some  circumstances,  and  users  need 
to  work  with  analysts  both  to  fully  understand  the  goals  of  the  survey  and  to  experiment 
with  appropriate  settings. 


17  For  example,  see  the  metadata  for  the  Puget  Sound  Lidar  Coalition  at 

http://Duaetsoundlidar.ess.washinaton.edu/lidardata/metadata.html.  or  for  the  Pennsylvania  PAMAP 
program  at  http://www.pamap.info/faq/lidar.htm#FAQ01 10. 
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4.2.5  Topographic  Break  Lines 

One  important  characteristic  of  lidar  is  that  this  technology  does  not  record  topographic 
break  lines  (Figure  13).  This  means  that  the  apparent  edges  of  features  cannot  always 
be  regarded  with  great  confidence.  The  absence  of  break  lines  can  have  serious 
impacts  on  the  creation  of  contour  lines  (especially  along  shorelines  or  edges  of  roads) 
and  on  the  operation  of  hydraulic  models.  In  applications  where  break  lines  are 
important,  these  can  be  acquired  through  ground  survey  or  other  appropriate  methods 
and  added  as  inputs  to  the  creation  of  the  DEM. 

Figure  12:  Hillshade  Display  Method  Examples 


Hilshade  image,  ArcGIS  default  settings,  former  Hillshade  image,  alternate  settings  showing  less 

Camp  Beale  ESTCP  demonstration  site.  The  dark  grey  area  but  also  less  contrast  on  the  craters 

default  settings  are  too  dark  in  some  areas  to  reveal 
the  ground  surface. 
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Hillshade  image,  “neutral”  hillshade  settings 


Hillshade  image,  contrast  boosted,  highlighting 
craters 


Figure  13:  Absence  of  Topographic  Break  Lines  in  Lidar-Derived  DEMs 


The  small  dam  is  visible  under  the  trees  in  the  lidar  surface  model.  However,  the  edges  of  the  dam  cannot 
be  precisely  determined  from  the  lidar  data  alone  since  there  is  no  guarantee  that  laser  signals  will  fall 
exactly  at  the  edges.  Former  Camp  Beale  ESTCP  demonstration  site. 


4.3  Lidar-Derived  Contour  Lines 

Contour  lines  also  are  a  typical  product  of  lidar  data,  and  contour  line  creation  introduces 
additional  technical  choices,  along  with  further  idealization  of  the  original  data.  Users 
are  often  surprised  that  lidar-based  contours  can  look  highly  irregular  and  jagged,  rather 
than  the  smooth  contours  that  are  usually  found  on  topographic  maps.  This  effect  is  a 
result  of  the  detailed  lidar  ground  surface  being  interpreted  through  computer  methods 
rather  than  a  human  operator  (Figure  14). 

Smooth  contour  lines  can  be  created  from  lidar  data  using  a  variety  of  methods; 
however,  these  should  be  approached  with  some  caution  and  adjusted  to  the  particular 
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project  needs.  Caution  is  required  because  computer-based  methods  for  creating 
smooth  contour  lines  may  remove  small  ground  features  that  may  be  important. 


4.4  Implications  for  Munitions  Sites 


Creation  and  display  of  DEMs  and  DSMs  for  military  munitions  sites  should  be  adapted 
to  the  detection  of  small  ground  features,  a  use  of  lidar  that  is  different  from  many  other 
applications  of  the  technology.  Site  managers  should  be  prepared  to  experiment  with 
appropriate  methods;  however,  general  approaches  should  include: 

■  Point  classification  methods  should  be  discussed  with  the  vendor’s  processing 
staff  and  adapted  to  maximize  the  number  of  points  included  in  the  ground 
surface  model 

■  Some  experimentation  should  be  conducted  to  determine  the  most  appropriate 
methods  for  interpolating  cell  elevation  values 

■  DEM  and  DSM  cell  sizes  should  be  kept  as  small  as  justifiable  by  the  density  of 
the  data  collected 


■  DEM  and  DSM  display  methods  should  be  adjusted  for  maximum  clarity 

Lidar  data  is  available  for  many  parts  of  the  country;  however,  the  existing  lidar  sets  are 
often  not  appropriate  for  munitions  investigations.  This  is  because  the  available  DEMs 
are  created  with  cells  that  are  too  large  to  detect  small  surface  features,  and  because 
the  underlying  lidar  data  is  generally  not  sufficiently  dense  to  justify  a  smaller  cell  size. 


Figure  14:  Contour  Lines  from  Lidar  Surface  Models 
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1  m  (black)  and  10  cm  (brown)  contour  lines  created  from 
DEM  after  application  of  ArcGIS  “filter”  command  (3 
passes).  Contour  lines  are  somewhat  smoothed. 


1  m  (black)  and  10  cm  (brown)  contour  lines  created  from 
DEM  after  application  of  ArcGIS  “filter”  command  (5 
passes).  Contour  lines  are  more  smoothed. 


33 


ESTCP  White  Paper:  Errors  in  Lidar  Data:  Implications  for 
Investigation  of  Military  Munitions  Sites 
Project  Number  07  E-MM2-012/MM-0737 


5  Published  Map  Accuracy  Standards 


Published  accuracy  standards  are  commonly  applied  to  products  derived  from  lidar, 
including  DEMs  and  contour  lines.  These  accuracy  standards  derive  historically  from 
map  accuracy  standards  for  photogrammetry.  In  photogrammetric  mapping,  such  as  for 
the  production  of  topographic  maps,  the  vertical  accuracy  of  the  printed  contour  lines  is 
largely  a  function  of  the  flying  height  of  the  aircraft  and  the  characteristics  of  the  mapping 
camera  used.  Early  map  accuracy  standards,  therefore,  consisted  of  required  flight 
altitudes  and  camera  specifications  to  create  contour  lines  of  a  desired  interval.  A 
standard  reference  describes  this,  for  example,  as:  “if  a  client  needed  2’  contours,  a 
typical  flying  height  for  acquisitions  of  aerial  photography  is  4,000  feet  above  mean 
terrain  when  using  a  mapping  camera  with  a  6”  focal  length:  if  a  client  needed  5-foot 
contours,  a  typical  flying  height  is  10,000  feet,  ”  (Maune  2001 ,  p  63). 

Non-photogrammetric  methods,  such  as  sonar,  lidar,  or  synthetic  aperture  radar,  are 
less  dependent  on  flight  altitude  to  determine  the  accuracy  of  the  data.  Map  accuracy 
standards  have  begun  to  evolve  in  response  to  the  emergence  of  these  technologies, 
and  this  evolution  is  still  underway.  Users  of  lidar  may  encounter  any  of  the  standards 
described  in  this  section. 

5.1  National  Map  Accuracy  Standard 

The  oldest  map  accuracy  standard  still  in  common  use,  the  National  Map  Accuracy 
Standard  (NMAS)  was  published  in  1947  by  the  US  Bureau  of  the  Budget18.  The  NMAS 
defined  horizontal  and  vertical  accuracies  for  contour  maps  with  a  published  scale  and 
contour  interval.  Vertical  accuracy  is  generally  defined  such  that  “not  more  than  10 
percent  of  the  elevations  tested  shall  be  in  error  more  than  one-half  the  contour  interval." 
However,  apparent  vertical  accuracy  may  be  decreased  based  on  permissible  horizontal 
error  for  a  map  of  that  scale. 

Lidar  vendors  are  often  required  to  meet  NMAS  accuracy  standards,  especially  where 
contour  lines  are  one  of  the  requested  deliverables.  However,  it  is  important  to  note  that 
the  NMAS  predated  the  development  of  DEMs,  and  it  does  not  contain  a  mechanism  for 
evaluating  the  accuracy  of  either  lidar  points  or  DEMs.  At  least  one  standard  reference 
recommends  that  NMAS  not  be  used  for  evaluating  and  reporting  the  vertical  accuracy 
of  DEMs  (Maune  2001). 

5.2  American  Society  of  Photogrammetry  and  Remote  Sensing  Standard 

In  1990,  the  American  Society  of  Photogrammetry  and  Remote  Sensing  (ASPRS) 
published  the  ASPRS  Interim  Accuracy  Standards  for  Large  Scale  Maps  (ASPRS  1989), 
with  the  expectation  that  these  standards  would  form  the  basis  for  revision  of  the  NMAS. 
The  ASPRS  Standard  postdates  the  introduction  of  DEMs  but  still  applies  to  graphic 
contour  maps  with  a  published  scale  and  contour  interval.  Like  the  NMAS,  vertical 
accuracy  may  be  adjusted  based  on  horizontal  accuracy. 


18  See:  http://egsc.usgs.gov/isb/pubs/factsheets/fs17199.html. 
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Unlike  the  NMAS,  the  ASPRS  Standard  defines  horizontal  and  vertical  accuracy  in  terms 
of  RMSE  from  checked  points,  using  check  surveys  of  a  higher  accuracy.  As  with  the 
NMAS,  the  ASPRS  Standard  is  most  appropriately  applied  to  contour  maps,  and 
standard  sources  recommend  that  it  not  be  used  for  evaluating  and  reporting  the 
accuracy  of  DEMs  (Maune  2001 ). 

5.3  National  Standard  for  Spatial  Data  Accuracy 

In  1998,  the  Federal  Geographic  Data  Committee  endorsed  and  published  a  set  of 
standards  that  included  the  National  Standard  for  Spatial  Data  Accuracy  (NSSDA).  The 
NSSDA  was  developed  to  evaluate  and  report  the  accuracy  of  digital  geospatial  data, 
including  DEMs  (Maune  2001).  For  both  horizontal  and  vertical  accuracy,  the  NSSDA 
establishes  accuracy  as  the  area  of  uncertainty  within  which  the  true  value  falls  95%  of 
the  time.  (The  area  of  uncertainty  is  circular  for  horizontal  uncertainty  and  linear  for 
vertical  uncertainty  [FGDC  1998]).  The  NSSDA  postdates  the  development  of  DEMs 
and  can  replace  the  NMAS  for  digital  geospatial  data  including  DEMs. 

Unlike  the  MNAS  and  ASPRS  Standard,  accuracy  is  computed  in  terms  of  ground 
distances  as  opposed  to  map  distances.  Reported  accuracy  values  reflect  all 
uncertainties,  including  those  introduced  by  geodetic  control  coordinates,  compilation, 
and  final  computation.  Apparent  vertical  errors  in  DEMs  are  not  offset  by  permissible 
horizontal  errors. 

Flowever,  the  NSSDA  does  not  define  threshold  accuracy  values,  stating  that  “Agencies 
are  encouraged  to  establish  thresholds  for  their  product  specifications  and  application 
and  for  contracting  purposes.”  This  lack  of  threshold  accuracy  values  may  account  for 
the  persistence  of  the  use  of  NMAS  and  ASPRS  Standard,  which  do  contain  threshold 
values  for  acceptable  accuracy. 

5.4  Federal  Emergency  Management  Agency 

Lidar  users  may  encounter  the  accuracy  standard  established  by  the  Federal 
Emergency  Management  Agency  (FEMA  2003).  The  FEMA  standard  is  a  variation  of 
the  NSSDA  with  an  agency-specified  accuracy  threshold,  and  with  an  emphasis  on 
contour  lines  for  flood  plain  mapping.  The  FEMA  standard  calls  for  DEMs  with  a 
minimum  5-m  post  spacing,  and  a  vertical  RMSE  of  15  cm,  evaluated  separately  for  all 
(normally  3-5)  land  cover  types  representative  of  the  floodplain  being  mapped. 

5.5  Lidar  Contour  Lines  and  Map  Accuracy  Standards 

The  creation  of  contour  lines  from  lidar  data  introduces  important  issues  regarding  the 
relationship  of  lidar  accuracy  and  density  to  the  contour  interval  chosen  and  the 
certification  of  contour  line  elevations.  As  a  general  rule,  smaller  contour  line  intervals 
require  lidar  data  that  is  denser  and  more  accurate. 

One  standard  source  for  guidance  on  accuracy  standards  for  contour  lines  and  other 
elevation  data  is  the  ASPRS  guidelines  (ASPRS  2004). 19  In  terms  of  setting  accuracy 


19  Online  at: 

http://www.asprs.org/society/committees/lidar/DownloadsA/ertical_Accuracy_Reporting_for_Lidar_Data.pdf. 
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standards  for  contour  lines,  ASPRS  points  out  that  contour  lines  are  produced  through  a 
series  of  technical  steps,  each  of  which  has  the  potential  to  introduce  error. 
Consequently,  contour  line  accuracy  should  be  specified  for  the  contours  themselves, 
and  not  be  confused  with  accuracy  of  the  lidar  points.  The  guidelines  conclude  that 
“Specifying  accuracy  of  the  final  product(s)  requires  the  data  producer  to  ensure  that 
error  is  kept  within  necessary  limits  during  all  production  steps.  ” 

Table  2  shows  the  ASPRS-recommended  general  correspondence  between  contour  line 
interval  and  data  accuracy. 


Table  2:  Recommended  Correspondence  Between  Contour  Line  Interval 

and  Data  Accuracy 


NMAS  Equivalent 
Contour  Interval 
(feet) 

NSSDA  RMSE(Z) 
(cm) 

NSSDA  Accuracy(Z) 
(cm) 

Reference  Data  Accuracy 
Required  for  “Tested  to 
Meet”  (feet) 

0.5 

0.15  ft  or  4.60  cm 

0.30  ft  or  9.10  cm 

0.10 

1 

0.30  ft  or  9.25  cm 

0.60  ft  or  18.2  cm 

0.20 

2 

0.61  ft  or  18.5  cm 

1.19  ft  or  36.3  cm 

0.40 

4 

1 .22  ft  or  37.0  cm 

2.38  ft  or  72.6  cm 

0.79 

5 

1.52  ft  or  46.3  cm 

2.98  ft  or  90.8  cm 

0.99 

10 

3.04  ft  or  92.7  cm 

5.96  ft  or  181.6  cm 

1.98 

ASPRS  recommends  that  in  contracting  for  lidar  data  production,  the  required  vertical 
accuracy  should  be  specified  in  terms  of  Accuracy(z)  in  the  third  column  of  the  table. 

That  is,  production  of  1-foot  contour  lines  would  require  a  vertical  accuracy  of  0.60  foot 
or  18.2  cm  20  Most  well-controlled  lidar  data  sets  should  at  least  meet  this  standard. 

However,  it  is  important  to  note  that  vertical  error  is  not  evenly  distributed  within  the  lidar 
data  set.  As  described  in  Section  2.5,  vertical  error  will  be  lower  in  flat,  open  areas  and 
higher  in  rugged  terrain  or  areas  of  greater  vegetation  cover.  Consequently,  in 
applications  where  contour  line  accuracy  is  critical,  it  may  be  appropriate  to  require 
reporting  of  lidar  accuracies  separately  for  different  ground  cover  categories.  This  would 
require  placement  of  appropriate  survey  controls  in  each  ground  cover  type. 

Vendors  contacted  generally  reported  procedures  that  agreed  with  the  ASPRS 
standards.  Vendors  reported  that  they  occasionally  received  requests  for  1-foot  contour 
lines.  To  comply  with  such  requests  required,  they  felt,  a  point  density  of  at  least  2  to  3 
points  per  square  meter  with  5  to  6  points  being  better,  and  with  control  points 
distributed  through  all  ground  cover  types.  They  further  reported  that  they  would  not 
create  1-foot  contour  lines  for  steep  or  very  uneven  ground  surfaces,  where  horizontal 
error  in  the  lidar  data  would  result  in  unacceptable  vertical  error. 

Creation  of  1-foot  or  smaller  contour  lines  is  possible  with  some  lidar  data  sets,  but 
presents  special  problems  and  ground  survey  may  be  more  appropriate.  Users  of  lidar 
data  occasionally  specify  accuracy  sufficient  to  create  0.5  foot  contour  lines.  At  the  time 
of  this  paper,  only  a  few  vendors  would  undertake  work  at  this  accuracy  level,  which  is 
accomplished  by  flying  at  extremely  low  altitudes  with  very  high  point  densities  the  most 
accurate  equipment  available. 


20 

Vertical  accuracy,  in  this  case,  would  be  determined  using  the  formula  Accuracy(z)  =  1.9800*RMSE(Z),  assuming 
normally  distributed  error.  See  ASPRS  Guidelines,  p.  3  (ASPRS  2004). 
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5.6  Implications  for  Munitions  Sites 

At  military  munitions  sites,  map  accuracy  standards  could  be  used  to  assure  the 
successful  spatial  integration  of  lidar  data  with  other  spatial  data  such  as  field  surveys, 
magnetometry  and  EMI  data.  However,  since  earlier  map  accuracy  standards  were 
designed  for  production  of  contour  lines  on  paper  maps,  it  would  be  most  appropriate  to 
use  a  more  modern  standard  such  as  the  NSSDA,  with  the  addition  of  a  specific 
accuracy  threshold  such  as  that  adopted  by  FEMA. 
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6  Summary  and  Implication  for  the  Use  of  Lidar  at  Munitions  Sites 


6.1  Lidar  in  Wide  Area  Assessment 

The  ESTCP  Wide  Area  Pilot  Program  demonstrated  that  lidar  which  met  its  stated 
accuracy  specifications  can  make  a  cost-effective  contribution  to  characterizing 
munitions  sites.  The  use  of  lidar  at  the  ESTCP  demonstration  sites  provided  corrections 
to  the  initial  conceptual  site  model  at  each  site  investigated,  including  locations  and 
revised  boundaries  for  munitions  response  sites  (MRSs).  At  the  ESTCP  demonstration 
sites,  features  with  elevation  differences  under  10  cm  were  successfully  detected, 
including  bombing  targets  and  potential  craters.  The  ESTCP  program  managers  used 
lidar  data  to  plan  the  deployment  of  follow-on  technologies,  and  integrated  lidar  data  with 
spatial  data  from  all  of  the  other  data  sources. 

6.2  Controlling  Lidar  Error 

Lidar  error  is  discussed  in  terms  of  the  lidar  points,  associated  use  of  GPS,  and  creation 
of  digital  models. 

6.2.1  Lidar  Points 

It  is  important  to  remember  that  the  objective  of  the  lidar  survey  is  not  to  detect  individual 
munitions  items,  but  rather  to  delineate  MRSs  and  to  focus  the  use  of  follow-on 
technologies  such  as  magnetometry  and  EMI.  Further,  the  features  being  identified, 
such  as  potential  craters,  are  larger  than  individual  ordnance  items  identified  by 
magnetometry  and  EMI.  Consequently,  the  level  of  positional  error  that  can  be  tolerated 
in  lidar  points  is  potentially  somewhat  higher  than  those  for  individual  ordnance  items, 
and  allows  the  use,  for  instance,  of  DEMs  and  DTMs  with  0.3  -1.0m  cell  sizes.  Based 
on  the  results  at  the  ESTCP  demonstration  sites,  it  appears  highly  likely  that  the 
vendors’  standard  guaranteed  accuracy  levels  are  sufficient  for  lidar  points. 

Controlling  error  of  the  lidar  points  is  primarily  a  function  of  the  vendor’s  quality  control 
methods.  With  few  exceptions,  accuracy  of  the  lidar  points  is  best  assured  by 
establishing  and  adhering  to  appropriate  contract  specifications  as  part  of  the  contract 
for  data  acquisition.  Specifications  for  horizontal  and  vertical  accuracy,  along  with  those 
for  flightline  to  flightline  integration,  can  be  verified  by  the  vendor  as  part  of  its  standard 
quality  assurance/quality  control  report,  and  can  be  independently  verified  by  end  users. 

However,  in  applications  where  higher  accuracy  is  required,  Government  end  users  may 
require  lower  flight  elevations  and  a  narrower  field  of  view,  at  some  increase  in  cost. 

6.2.2  GPS 

GPS  error  has  implications  for  the  accuracy  of  the  lidar  points  themselves,  and  GPS  is 
also  used  to  establish  calibration  points  and  locations  of  features  in  the  field. 

Lidar  points.  As  long  as  GPS  error  does  not  exceed  assumed  values  quoted  by 
vendors,  the  resulting  lidar  data  should  be  within  contracted  accuracy  specifications. 
Monitoring  GPS  quality  and  other  factors  affecting  positional  accuracy  should  be  part  of 
the  vendor’s  daily  quality  assurance/quality  control  procedures. 
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Calibration  and  quality  control.  Standard  practice  in  establishing  calibration  points 
includes: 

■  Using  survey-grade  GPS  operated  by  qualified  staff 

■  Choosing  flat  sites  to  minimize  the  effects  of  slope  between  surrounding  control 
point  and  nearest  lidar  points 

■  Choosing  open  sky  sites,  which  are  essential  to  GPS  quality 

■  Choosing  calibration  points  in  all  major  terrain  types 

Field  investigations.  In  most  field  work,  resource-grade  GPS,  with  its  sub-meter 
accuracy,  should  be  able  to  locate  munitions-related  ground  features  with  sufficient 
accuracy  to  compare  with  the  lidar  data,  as  long  as  open-sky  conditions  are  present.  At 
vegetated  sites,  resource-grade  GPS  may  not  provide  positions  of  sufficient  accuracy, 
and  other  methods  may  be  necessary.  . 

6.2.3  Digital  Models 

In  the  creation  of  digital  models,  vendors  and  Government  end  users  have  several 
options  to  assure  that  DEMs  and  DTMs  best  serve  the  needs  of  the  site  investigations: 

■  Vendors  should  be  required  to  meet  tight  calibration  standards  to  minimize 
elevation  differences  that  may  result  in  over-classification  of  non-ground  points. 
Optimum  point  classification  methods  for  munitions  sites  have  not  yet  been 
developed;  however,  the  appropriate  principle  is  to  retain  the  largest  number  of 
points  possible  in  the  ground  classification,  even  at  the  cost  of  including  some 
additional  surface  noise. 

■  In  creating  DEMs  and  DTMs,  vendors  and  Government  end  users  should  use  the 
smallest  cell  size  consistent  with  data  density. 

■  Vendors  and  end  users  should  consider  experimenting  with  interpolation 
methods. 

In  practice,  creating  DEMs  and  DTMs  is  more  conveniently  accomplished  by 
Government  end-users  in-house,  rather  than  having  the  vendor  deliver  them,  since 
creation  in-house  can  allow  for  experimentation  with  alternative  methods.  However,  this 
requires  that  Government  end-users  have  sufficient  software  tools  and  training  to 
accomplish  these  tasks.  Creating  DEMs  and  DTMs  in-house  can  be  challenging, 
especially  with  high-density  data  sets  where  the  number  of  points  is  very  large.  In  such 
cases  it  may  be  appropriate  to  work  with  sample  data  sets  to  determine,  in  consultation 
with  vendors,  the  appropriate  specifications  for  these  products,  and  then  request  that  the 
vendor  create  and  deliver  the  final  products. 
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Attachment  A 

Formulas  for  Calculating  Horizontal  and  Vertical  Root  Mean  Square  Error 

The  following  formulas  are  used  to  compute  horizontal  and  vertical  root  mean  square 
error  (RMSE)  and  horizontal  and  vertical  accuracy  at  the  95%  confidence  level.  For 
example,  these  formulas  are  used  in  reporting  accuracy  based  on  the  map  accuracy 
standards  such  as  the  National  Standard  for  Spatial  Data  Accuracy  (NSSDA),  discussed 
in  Section  4.3. 

Horizontal  Root-Mean-Square  Error  and  Accuracy 


RMSEX  —  Sqrt[X(X  data  I  X  check  I  )  /n] 
RMSEy  =  sqrt[X(y  data  i  -  y  check  i  )2/n] 


Where: 


X  data  i,  y  data  i  are  the  coordinates  of  the  Ith  check  point  in  the  dataset 

X  check  i,  y  check  i  are  the  x  coordinates  of  the  Ith  check  point  in  the  independent 
source  of  higher  accuracy 

n  is  the  number  of  check  points  tested 

I  is  an  integer  ranging  from  1  to  n 

Horizontal  error  at  point  I  is  defined  as: 

Sqrt[(x  data  I  —  X  check  l)2  +  (y  data  I  Y  check  l) 

Horizontal  RMSE  (RMSEr)  is: 

RMSEr  —  sqrt[£((x  data  I  —  X  check  l)  (Y  data  I  —  Y  check  l)  )/ri] 

=  sqrt[RMSEx2  +  RMSEy2] 

Computing  accuracy  according  to  the  NSSDA  when  RMSEX  =  RMSEy 

RMSEr  =  sqrt(2  *  RMSEX2)  =  sqrt(2  *  RMSEy2) 

=  1.4142  *RMSEX=  1.4142*  RMSEy 

In  the  use  of  these  formulas  “It  is  assumed  that  systematic  errors  have  been  eliminated 
as  best  as  possible.  If  error  is  normally  distributed  and  independent  in  each  the  x-  and 
y-component  and  error  for  the  x-component  is  equal  to  and  independent  of  error  for  the 
y-component,  the  factor  2.4477  is  used  to  compute  horizontal  accuracy  at  the  95% 
confidence  level”  (Greenwalt  and  Schultz  1968).  When  these  conditions  apply, 
horizontal  accuracy,  Accuracyr,  may  be  computed  by  the  formula: 

Accuracy,- =  2.4477  *  RMSEX  =  2.4477  *  RMSEy  =  2.4477  *  RMSEr/14142 

Accuracyr  =  1.7308  *  RMSEr 
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Vertical  Root-Mean-Square  Error  and  Accuracy 


RMSEz  =  Sqrt[X(z  data  I  -  Z  check  I  )2/n] 


Where: 


z  data  i,  is  the  vertical  coordinate  of  the  Ith  check  point  in  the  dataset 

z  check  i>  is  the  vertical  coordinates  of  the  Ith  check  point  in  the  independent  source 
of  higher  accuracy 

N  is  the  number  of  points  being  checked 
I  is  an  integer  ranging  from  1  to  n 

As  with  horizontal  accuracy,  “It  is  assumed  that  systematic  errors  have  been  eliminated 
as  best  as  possible.  If  vertical  error  is  normally  distributed,  the  factor  1 .9600  is  applied 
to  compute  linear  error  at  the  95%  confidence  level”  (Greenwalt  and  Schultz  1968). 
Therefore,  vertical  accuracy,  Accuracyz,  reported  according  to  the  NSSDA  shall  be 
computed  by  the  following  formula: 

Accuracyz  =  1 .9600  *  RMSEZ 

Reference 

Greenwalt,  C.R.  and  M.E.  Schultz.  1968.  Principles  and  Error  Theory  and  Cartographic 
Applications.  ACIC  Technical  Report  No.  96.  St.  Louis,  Mo.  Aeronautical  Chart 
and  Information  Center,  U.S.  Air  Force. 
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1  Introduction  and  Summary  of  Key  Findings 


1.1  Introduction 

Many  millions  of  acres  of  Department  of  Defense  (DoD)  lands  are  potentially  contaminated  with 
military  munitions  or  their  components.  On  the  majority  of  these  sites,  munitions  are 
concentrated  in  specific  ranges  and  training  areas.  Locating  the  site  of  contamination  can  be 
difficult,  in  part  because  historical  records  are  often  incomplete  or  inaccurate. 

Between  2005  and  2007,  the  Environmental  Security  Technology  Certification  Program 
(ESTCP)  conducted  a  pilot  program  to  test  the  effectiveness  of  a  multi-technology  approach  to 
unexploded  ordnance/munitions  and  explosives  of  concern  (UXO/MEC)  wide  area  assessment 
(WAA).  The  program  included  the  use  of  light  detection  and  ranging  (lidar),  orthophotography, 
helicopter  magnetometry,  towed-array  magnetometry,  and  statistically-based  transect  design,  in 
a  comprehensive,  sequential  approach.  The  first  phase  of  this  program  was  carried  out  at  three 
desert  sites  containing  little  or  no  vegetation  and  few  non-military  land  uses.  Subsequently,  a 
second  phase  of  the  pilot  program  was  added,  including  two  new  sites:  the  Former  Camp  Beale 
site  near  Marysville,  California,  and  the  Toussaint  River  site  near  Lake  Erie.  The  Former  Camp 
Beale  site  has  more  varied  vegetation  cover  and  land  use  types;  the  Toussaint  River  site  is  a 
shallow-water  site. 

This  white  paper  is  a  follow-up  to  the  investigation  of  the  ESTCP  demonstration  sites  where 
lidar  data  was  collected.  At  these  sites,  lidar  met  the  objectives  of  the  demonstration: 

■  Lidar  was  used  to  identify  munitions  response  sites  (MRS)  such  as  bombing  targets, 
berms,  and  firing  points 

■  The  results  of  the  lidar  investigation  were  used  to  supplement  or  correct  the  initial 
conceptual  site  model  (CSM)  based  on  historical  records  review 

■  Lidar  data  provided  useful  input  to  subsequent  phases  of  site  investigation,  including 
information  on  slope  and  vegetation  cover 

Further  examination  of  the  lidar  data  for  the  ESTCP  demonstration  sites  showed  that  there  may 
be  potential  to  improve  the  resolution  of  the  lidar  surface  models  by  improving  the  way  lidar 
returns  are  classified.  An  important  part  of  the  use  of  lidar  is  the  classification  of  laser  signals 
as  returning  from  ground  or  non-ground  surfaces.  At  all  of  the  ESTCP  sites  where  lidar  was 
used,  some  of  the  lidar  points  classified  as  returning  from  non-ground  surfaces  appeared  to  be 
returns  from  ground.  If  classification  could  be  modified  to  include  more  of  these  points  in  the 
ground  surface  model,  the  ground  surface  could  be  portrayed  with  higher  resolution,  and  the 
ability  to  detect  small  features  could  be  increased. 

A  preliminary  draft  of  this  paper  presented  the  results  of  re-analysis  of  the  lidar  data  for  the 
Former  Camp  Beale  demonstration  site.  Following  review  by  ESTCP  of  the  methodology  used, 
data  from  six  additional  sites  were  analyzed,  and  results  are  presented  in  this  draft. 

1 .2  Objectives 

This  white  paper  examines  the  methodologies  in  use  for  classification  of  laser  returns,  the 
possibility  that  some  laser  returns  are  being  incorrectly  classified  as  non-ground,  and  potential 
benefits  of  including  more  of  the  lidar  points  in  the  digital  elevation  model  (DEM)  representing 
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the  ground  surface.  The  findings  and  conclusions  of  this  paper  are  intended  to  contribute  to 
contract  specifications  and  analysis  methods  that  can  be  used  by  Government  site  managers 
who  are  planning  to  use  lidar. 

Key  questions  examined  include: 

■  How  do  lidar  vendors  distinguish  reflections  from  the  ground  or  other  objects 
such  as  vegetation? 

■  What  point  classification  results  were  achieved  at  the  ESTCP  demonstration 
sites? 

■  Would  different  classification  methods  improve  the  resulting  surface  models  and 
lidar  feature  detection? 

■  If  so,  how  great  is  the  difference  and  how  much  extra  effort  is  required? 

1.3  Summary  of  Key  Findings 

At  the  ESTCP  demonstration  sites,  a  significant  number  of  laser  returns  from  the  ground  surface 
appear  to  have  been  classified  as  non-ground.  Visual  examination  showed  that  some  returns 
from  paved  surfaces  were  classified  as  non-ground.  Samples  from  a  total  of  1 1  data  sets  from 
the  four  sites  where  lidar  was  collected  showed  that  from  15  to  over  95  percent  of  the  ground 
returns  may  have  been  classified  as  non-ground  depending  on  the  sample  area  examined. 

The  phenomenon  of  laser  returns  from  the  ground  being  classified  as  non-ground  appears  to 
result  from  the  operation  of  the  standard  TerraSolid  software  used  for  point  classification, 
especially  as  it  interacts  with  small  elevation  differences  between  lidar  points.  The  TerraSolid 
software  uses  a  sophisticated  set  of  automated  algorithms  for  classifying  laser  returns.  An  initial 
surface  model  is  first  created  using  the  lowest  points  in  the  data  set,  after  which  points  are 
added  iteratively  if  they  meet  pre-set  parameters  intended  to  prevent  the  addition  of  points  that 
form  “spikes”  characteristic  of  vegetation  or  buildings.  An  important  parameter  in  this  process  is 
the  angle  between  each  new  point  examined  and  the  nearest  already-included  point. 

Point  classification  results  can  be  influenced  by  increasing  either  the  spatial  discrepancies 
between  the  lidar  points,  as  when  flight  lines  are  not  tightly  calibrated,  or  by  increasing  the 
density  of  lidar  returns.  When  the  density  of  lidar  returns  increases,  the  laser  returns  are  closer 
together  and  the  angle  formed  by  any  new  point  will  be  larger,  even  when  the  new  point  is  only 
slightly  above  the  already-modeled  surface.  When  laser  returns  are  sufficiently  close,  this 
increase  in  angle  can  be  large  enough  that  very  small  elevation  variations,  within  the  noise  level 
of  the  equipment,  can  cause  points  to  be  classified  as  non-ground. 

In  most  lidar  surveys,  classifying  numerous  points  as  non-ground  is  acceptable  and  even 
desirable.  This  is  because  the  resulting  ground  surface  will  be  smoother  and  points  will  be 
misclassified  only  in  areas  where  the  overall  density  is  higher  than  needed  to  generally 
characterize  the  ground  surface.  However,  this  method  may  not  be  appropriate  where  the 
objective  of  the  survey  is  to  detect  small  objects. 

For  the  six  sites  and  1 1  lidar  data  sets  examined,  the  percentage  of  laser  returns  classified  by 
the  vendor  as  non-ground  was  calculated  for  a  series  of  test  areas.  Where  possible,  test  areas 
were  located  on  paved  surfaces  where  all  “non-ground”  classifications  could  be  assumed  to  be 
artifacts  of  the  classification  process.  On  sites  with  no  paved  surfaces,  test  areas  were  placed 
in  areas  of  sparse  and  uniform  vegetation. 
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At  the  four  ESTCP  sites  (nine  data  sets),  the  percentage  of  laser  returns  classified  as  non¬ 
ground  ranged  from  approximately  1 5  percent  to  over  95  percent.  For  all  but  one  of  the  data 
sets,  the  rate  at  which  points  were  classified  as  non-ground  increased  very  closely  with 
increasing  overall  density  of  lidar  points,  and  visual  inspection  showed  that  points  were 
classified  as  non-ground  at  much  greater  rates  in  areas  of  flight  line  overlap.  Further,  the 
operation  of  the  classification  software  held  the  density  of  ground  returns  to  a  maximum  of 
around  2.5  points  per  square  meter  (pts/m2),  regardless  of  the  overall  point  density  (which 
ranged  as  high  as  41  pts/m2). 

At  the  two  non-ESTCP  sites  (two  data  sets),  the  overall  data  density  was  lower  than  for  the 
ESTCP  sites  (between  0.78  and  1 .55  pts/m2  compared  to  between  1 .69  and  41 .50  pts/m2  for 
the  test  areas  at  the  ESTCP  sites).  Both  non-ESTCP  test  areas  included  paved  roads,  and 
examination  of  the  data  showed  that  the  rate  of  classification  as  non-ground  on  road  surfaces 
was  much  lower  than  for  the  ESTCP  sites,  between  3.34  and  15.36  percent.  The  lower  rate  of 
misclassification  may  be  a  result  of  the  overall  lower  point  density  which  resulted  in  greater 
distances  between  the  laser  returns.  However,  some  laser  returns  from  paved  surfaces  were 
still  classified  as  non-ground,  and  the  mechanism  appeared  to  be  the  same  as  at  the  ESTCP 
sites. 

Classification  errors  of  this  kind  can  be  partially  eliminated  by  lowering  the  degree  of  spatial 
displacement  in  the  laser  returns,  both  between  one  point  and  the  next  and  between  one  flight 
line  and  the  next.  Vendors  contacted  commented  that  the  primary  emphasis  in  reducing  the 
amount  of  spatial  displacement  should  be  to  achieve  the  best  possible  calibration  of  the  data 
set,  especially  from  one  flight  line  and  the  next.  However,  some  degree  of  displacement  is 
inevitable  due  to  site  conditions  and  instrument  accuracy  limitations,  and  local  areas  of  high 
density  can  occur  for  other  reasons  than  flight  line  overlap.  Lidar  surveys  aimed  at  detecting 
small  surface  features  should  therefore  also  consider  adjustments  to  the  classification  methods 
used. 

The  TerraSolid  software  includes  numerous  settings  that  can  be  varied  at  will  to  increase  the 
number  of  points  classified  as  ground.  Modification  of  these  settings  should  not  result  in 
increased  time  or  effort.  Potential  modifications  range  from  specific  changes  to  individual 
classification  parameters  to  bulk  classification  of  all  laser  returns  within  a  given  distance  of  the 
original  modeled  surface  as  ground.  Classification  could  be  modified  either  to  directly  classify 
more  points  as  ground  returns  or  to  establish  a  new  classification  for  “low”  lidar  points  in 
addition  to  the  vendor’s  original  ground  classification. 

Notwithstanding  the  relative  ease  of  changing  the  classification  parameters,  a  change  of 
classification  is  valuable  only  if  it  results  in  enhanced  feature  detection.  A  test  reclassification 
was  performed,  using  a  series  of  59  test  areas  at  the  Former  Camp  Beale  site.  New  surface 
models  were  created  using  all  laser  returns  within  0.4  meters  (m)  of  the  ground  surface.  This 
value  was  chosen  as  a  means  to  include  the  maximum  number  of  additional  points  for  an  initial 
determination  of  increased  ability  to  reveal  features,  with  the  understanding  that  both  low 
vegetation  and  noise  also  would  be  included.  The  test  areas  all  included  ground  features  visible 
in  the  initial  surface  model  based  on  the  vendor’s  classifications,  typically  large  depressions.  All 
but  three  of  these  features  were  under  partial  or  full  tree  cover. 

The  surface  models  from  the  vendor’s  classification  were  compared  to  those  using  the 
reclassified  points.  At  21  of  the  test  areas  (36  percent),  the  new  surface  models  showed 
additional  potential  features.  These  features  were  all  small  and  poorly-defined,  and  most  could 
not  be  conclusively  identified.  However,  these  additional  potential  features  did  indicate  areas 
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for  further  investigation.  The  larger  features  in  the  surfaces  created  with  the  reclassified  points 
also  were  somewhat  better  defined  in  the  reclassified  surfaces. 

As  a  second  test,  the  size  of  the  grid  cells  in  the  surface  models  for  nine  test  areas  was 
changed  from  1  m  to  0.3  m  (1  foot),  using  both  the  vendor’s  original  classifications  and  the 
reclassified  points.  The  smaller  grid  cells  yielded  a  more  sharply-defined  ground  surface.  The 
reclassified  surfaces  showed  additional  small  features  in  a  manner  similar  but  not  identical  to 
the  1  m  cell  sizes,  with  the  smaller  cell  sizes  showing  more  additional  features. 

As  a  third  test,  new  surfaces  were  created  using  a  different  approach  to  surface  model  creation. 
At  all  of  the  ESTCP  demonstration  sites,  surface  models  were  created  by  first  creating  a 
Triangulated  Irregular  Network  (TIN)  using  every  laser  return  as  the  vertex  of  the  triangles. 
Surface  models  were  then  created  from  the  TINs.  This  method  was  chosen  since  it  uses  every 
available  laser  return  to  create  the  initial  model  from  which  the  surfaces  are  derived.  The 
alternative  to  this  method  is  to  use  one  of  several  methods  to  mathematically  interpolate 
surfaces  directly  from  the  points.  Interpolated  surfaces  were  created  for  three  test  areas,  using 
both  1  m  and  0.3  m  cell  sizes.  At  two  of  the  three  test  areas,  the  interpolated  surfaces  showed 
more  features  than  the  TIN-derived  surfaces.  The  reason  for  this  difference  is  not  clear,  and 
further  investigation  would  be  warranted. 

Finally,  the  1 .5  m  and  0.3  m  test  craters  at  the  Former  Camp  Beale  site  were  examined,  using 
the  models  created  from  the  vendor-classified  points  and  the  reclassified  points.  Since  the  test 
craters  were  located  in  an  area  with  no  tall  vegetation,  the  reclassified  points  included  all  of  the 
lidar  points.  The  more  dense  data  showed  the  test  craters  more  clearly,  and  included  additional 
noise.  However,  no  new  features  were  visible  using  the  full  data  set,  and  in  neither  case  were 
the  smallest  (0.30  m)  test  craters  seen. 

These  findings  lead  to  the  following  recommendations  for  subsequent  use  of  lidar  at  munitions 
sites: 


■  Lidar  remains  a  useful  tool  in  WAA,  and  should  be  included  where  appropriate  to 
the  site. 

■  Vendors  should  be  asked  to  describe  potential  point  classification  approaches 
prior  to  each  survey,  and  should  be  requested  to  adjust  classification  parameters 
depending  on  the  objectives  of  the  survey. 

■  Vendors  should  be  required  to  deliver  the  full  lidar  data  set,  including  all  points 
classified  as  non-ground  returns,  to  the  Government.  If  all  points  are  delivered, 
Government  land  managers  can  evaluate  the  classification  methods  used  and 
can  experiment  with  other  approaches  to  classification  as  warranted. 

■  Government  land  managers  may  consider  requesting  an  additional  classification 
from  the  vendor,  including  more  potential  ground  returns.  This  would  allow  the 
vendor’s  initial  classifications  to  be  used  for  applications  where  achieving  a 
smooth  ground  surface  was  more  important  (such  as  creating  contour  lines), 
while  using  the  “enhanced”  classifications  for  investigation  of  munitions-related 
features. 
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2  Overview  of  Lidar  Data  Processing  Steps 

This  section  outlines  the  components  of  the  lidar  data  collection  system  and  the  steps  through 
which  the  original  sensor  output  is  converted  to  lidar  x,  y,  z  locations.  This  information  is  not 
intended  to  be  comprehensive,  and  is  included  as  background  to  the  discussion  of  point 
classification  methods.  This  section  is  based  on  published  research,  user  manuals  for  standard 
software  packages,  and  interviews  with  vendors. 

2.1  Collecting  Lidar  Data  and  Creating  Lidar  Points 

2.1.1  Sensor  Components 

In  order  to  establish  the  location  from  which  the  laser  signal  is  reflected,  the  lidar  system  must 
answer  two  basic  questions:  1 )  where  is  the  aircraft  located  and  2)  what  is  the  distance  and 
angle  to  the  point  of  reflection  of  the  laser  pulse? 

Locating  the  aircraft.  The  position  of  the  aircraft  is  determined  using  the  Global  Positioning 
System  (GPS)  and  Inertial  Navigation  System  (INS). 

■  The  GPS  samples  the  aircraft  location  at  regular  intervals  (one  to  ten  times  per 
second),  based  on  received  signals  from  a  minimum  of  four  orbiting  navigation 
satellites.  GPS  position  inaccuracies  are  due  to  errors  in  calculating  the  exact 
time,  the  ranges  to  the  individual  satellites  and  noise.  The  standard  method  to 
limit  the  effects  of  correlated  errors  (e.g.,  propagation  delays  that  will  be  similar 
for  reasonably  close  points)  in  the  GPS  signal  is  to  establish  an  independent 
static  GPS  base  station  at  a  known  location  to  provide  error  correction 
information.  If  sufficiently  close,  the  reference  station  (static)  will  experience  the 
same  propagation  and  timing  errors  as  the  GPS  on  board  the  aircraft  (kinematic). 
Because  the  reference  is  fixed  and  its  location  is  known,  correlated  GPS  errors 
can  be  isolated  from  the  signal,  a  solution  developed,  and  the  solution  applied  to 
kinematic  GPS  data  in  post  processing  to  help  refine  the  position  of  the  aircraft. 
(Sky  Research  2009) 

■  The  INS  calculates  the  position  of  the  aircraft  between  GPS  locations.  The 
primary  input  to  the  INS  is  from  the  Inertial  Measurement  Unit  (IMU).  The  IMU 
works  by  detecting  the  changes  to  rates  of  acceleration,  along  with  rotational 
attributes  such  as  pitch,  roll,  and  yaw.  The  IMU  generally  contains  three 
accelerometers  and  three  gyroscopes,  each  placed  in  orthogonal  positions  so 
that  data  is  collected  in  all  three  planes.  Current  IMUs  record  these  changes  at 
200  hertz  (Hz)  (200  samples  per  second).  The  IMU  adds  a  small  error 
component  in  each  plane.  This  error  results  from  the  accumulation  of  small 
errors  as  positions  are  continually  re-calculated.  These  small  errors  accumulate 
until  the  aircraft’s  position  is  updated  from  the  GPS. 

Locating  the  laser  pulse.  Lidar  systems  measure  the  time  of  return  of  a  laser  pulse.  The  laser 
system  consists  of  a  laser,  a  receiver,  and  a  mirror  that  directs  each  pulse  toward  the  ground 
surface.  Lidar  systems  typically  use  lasers  with  wavelengths  of  1 ,000  to  1 ,500  nanometers. 
Pulses  are  directed  to  the  ground  by  a  mirror  system,  which  may  be  rotating  or  oscillating, 
depending  on  the  manufacturer. 
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The  receiver  is  a  passive  device  that  is  tuned  to  the  frequency  of  the  laser,  which  records  a 
signal  when  the  amplitude  of  light  in  that  frequency  exceeds  a  threshold  value.  The  sensor  also 
records  the  energy  level  of  the  return,  which  is  referred  to  as  its  intensity  value.  The  threshold 
energy  value  that  will  trigger  a  return  is  typically  set  by  the  manufacturer  and  is  not  adjustable. 

The  sensor  can  record  multiple  returns  from  each  laser  pulse.  In  vegetated  conditions,  part  of 
the  laser  return  may  be  reflected  from  branches  and  other  sub-canopy  features,  leaving  the 
remainder  of  the  signal  to  be  reflected  from  the  ground  surface.  The  limitation  of  the  multi¬ 
return  capability  is  that  there  must  be  sufficient  time  between  returns  for  the  sensor  to  reset. 

This  time  is  generally  between  three  and  20  nanoseconds  depending  on  the  sensor  used,  a 
distance  of  between  0.5  m  and  3  m.  Thus,  multiple  returns  cannot  be  recorded  in  low 
vegetation. 

A  relatively  recent  development  in  laser  technology  is  the  analog  or  “full  wave  form”  laser 
receiver.  In  contrast  to  the  approach  described  above,  the  analog  receiver  records  a  continuous 
level  of  energy  values  once  the  amplitude  passes  the  threshold  value.  The  shape  of  this  energy 
return  curve  can  be  analyzed  and  multiple  return  values  derived  in  a  more  interactive  manner. 
Analog  receivers  create  larger  data  sets  than  traditional  sensors,  and  more  advanced  software 
is  needed  to  process  the  output.  Analog  sensors  are  typically  used  for  analysis  of  vegetation 
rather  than  modeling  the  ground  surface  and  appear  to  offer  little  advantage  at  munitions  sites. 

The  final  components  of  the  sensor  system  are  the  power  source,  the  hardware-  and  software- 
based  control  system,  and  the  data  storage  equipment  (consisting  of  multiple  high-speed  hard 
drives). 

2.1.2  System  Settings 

The  lidar  system  has  some  parameters  that  can  be  varied,  all  within  the  equipment 
specifications  of  each  manufacturer.  These  inSclude: 

■  Power  level.  The  power  of  each  laser  pulse  is  adjustable  up  to  the  maximum 
power  of  the  system. 

■  Pulse  rate.  The  maximum  pulse  rate,  or  the  number  of  laser  signals  per  second, 
has  increased  steadily  as  the  technology  has  evolved,  from  four  to  10  kHz  in  the 
mid-1990s  to  as  much  as  250  kHz  at  the  time  of  this  report.  Pulse  rates  are 
typically  adjustable. 

■  Mirror  speed.  The  oscillation  or  rotation  speed  of  the  mirror  is  adjustable 
independently  of  the  laser  pulse  rate. 

■  Scan  angle.  The  maximum  scan  angle  of  the  mirror  off  nadir  is  adjustable.  In 
combination  with  the  aircraft  altitude,  the  scan  angle  controls  the  width  of  the 
data  collection  swath. 

Vendors  report  that  the  primary  parameters  that  are  varied  are  pulse  rate  and  scan  angle. 

Flight  altitude  and  air  speed  determine  the  average  along-track  point  spacing,  while  scan  and 
flight  altitude  determine  the  average  across-track  point  spacing. 

The  relationship  of  laser  pulse  rate  to  system  performance  is  not  always  straightforward.  This  is 
because  the  power  system  of  the  laser  is  finite,  and  as  a  result,  as  the  pulse  rate  is  increased, 
each  individual  laser  pulse  will  necessarily  have  less  power.  It  is  possible  to  design  a 
combination  of  flight  altitude  and  pulse  rate  that  will  cause  the  return  signal  to  be  too  weak  to 
trigger  the  receiver  and  record  a  return.  This  limitation  is  especially  notable  in  vegetated 
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conditions,  where  the  success  of  the  survey  will  depend  on  recording  multiple  returns.  Multiple 
returns  have  less  power  since  they  reflect  only  part  of  the  laser  pulse.  In  practice,  vendors  will 
adjust  power  levels,  pulse  rates,  and  fight  altitude  to  maximize  the  performance  of  the  system. 
Numerous  low-intensity  returns  or  failing  to  get  returns  over  less  reflective  surfaces  such  as 
blacktop  during  data  collection  may  indicate  that  power  should  be  increased,  altitude  lowered,  or 
pulse  rate  reduced. 

2.1.3  Sensor  Output 

The  output  of  the  sensor  system  is  a  series  of  data  values  recorded  by  the  GPS,  INS,  mirror 
system,  and  receiver.  These  data  values  record  the  action  of  each  of  these  components,  each 
with  a  time  record.  The  GPS  and  INS  records  allow  the  calculation  of  aircraft  location,  the 
mirror  system  records  the  angle  of  each  laser  pulse,  and  the  receiver  records  the  time  and 
intensity  of  each  return.  Timestamps  for  each  record  allow  the  data  to  be  assembled  and  the  x, 
y,  and  z  locations  that  become  the  lidar  points  created. 

2.1.4  Calibration  and  Data  Validation 

Calibration.  System  calibration  begins  just  prior  to  the  data  collection  flights.  Test  flights  are 
conducted  over  calibration  features  with  known  dimensions.  Flights  are  conducted  at  several 
headings.  On  return  to  the  office,  the  results  of  these  test  fights  are  used  to  establish  correction 
values  for  pitch,  roll,  and  yaw  so  that  the  features  remain  constant  at  all  headings.  In  a  well- 
calibrated  data  set,  the  average  vertical  offset  between  overlapping  flight  lines  should  be  at  or 
near  the  inherent  noise  level  of  the  data,  typically  around  3-4  cm  at  one  sigma  (Sky  Research, 
2009). 

Data  validation.  During  the  mission,  a  set  of  surveyed  validation  points  is  established  on  the 
ground  within  the  data  collection  area.  These  points  are  used  in  post  processing  to  vertically 
adjust  the  laser  data  to  known  ground  heights.  Establishing  a  higher  number  of  control  points 
will  in  most  cases  increase  accuracy. 

As  one  indicator  of  overall  data  quality,  the  variance  of  the  laser  data  is  calculated  by  comparing 
the  height  from  a  set  of  individual  validation  points  with  the  height  of  the  nearest  laser  return.  In 
some  cases,  a  high  degree  of  variance  may  be  an  indicator  of  poor  data  processing.  Terrain 
setting,  quality  of  ground  survey  points,  location  of  ground  survey  targets  (flat  ground  vs. 
uneven  ground),  and  density  of  the  laser  data  points  are  all  factors  that  may  affect  this  value  as 
well.  It  is  therefore  important  that  great  care  be  taken  when  collecting  data  for  validation  points 
and  that  the  location  of  these  points  be  chosen  carefully  (Sky  Research,  2009). 

The  number  of  surveyed  points  collected  varies  considerably  between  vendors  and  there 
appears  to  be  no  industry  standard.  Some  vendors  establish  three  to  four  static  survey  points; 
others  supplement  static  points  with  vehicle-mounted  GPS  surveys  on  roads  in  the  study  area; 
others  use  remote-controlled  rovers  to  establish  up  to  many  hundreds  of  static  survey  points. 

On  return  to  the  office,  the  lidar  data  is  compared  to  the  surveyed  points  to  validate  the  lidar 
data  values  by  comparing  the  height  of  the  surveyed  points  with  the  height  of  the  nearest  laser 
return.  Sources  of  vertical  error  in  the  laser  data  include  changes  in  atmospheric  conditions, 
residual  GPS  errors,  noise  inherent  to  the  apparatus,  and  calibration.  The  lidar  data  can  be 
adjusted  to  achieve  a  “best  fit”  to  the  control  points;  some  vendors  commented  that  failure  to  do 
so  may  result  in  vertical  accuracy  errors  of  up  to  0.5  meter  (Sky  Research,  2009). 
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A  high  degree  of  variance  between  lidar  data  and  survey  points  may  indicate  poor  data 
calibration.  If  calibration  values  are  incorrectly  established,  the  lidar  values  and  the  surveyed 
values  will  not  match.  If  the  lidar  data  is  not  consistent  throughout  the  site,  the  pitch,  roll,  and 
yaw  values  would  then  be  examined  and  recalculated,  or  other  potential  sources  of  error 
examined.  However,  terrain  setting,  quality  of  ground  survey  points,  location  of  ground  survey 
targets  (flat  ground  vs.  uneven  ground),  and  density  of  the  laser  data  points  are  all  factors  that 
may  affect  this  value  as  well.  It  is  therefore  important  that  validation  points  be  located  carefully, 
preferably  on  smooth,  flat  areas  where  terrain  variation  will  not  affect  the  comparison  between 
lidar  elevations  and  validation  point  elevations. 

2.1 .5  Exporting  the  Lidar  Values 

Once  the  data  are  calibrated  and  validated,  x,  y,  z  values  can  be  exported  to  various  formats 
that  can  be  read  by  GIS,  computer-aided  drafting  (CAD),  or  geophysical  programs.  Export  can 
take  place  before  or  after  classification,  and  can  include  any  of  the  data  collected  with  the 
return,  such  as  the  GPS  time,  intensity,  scan  angle,  and  flight  line  number  (if  this  is  recorded). 

2.2  Classifying  the  Lidar  Points 

2.2.1  The  Problem:  Distinguishing  Ground  Returns  from  Non-Ground  Returns 

In  all  lidar  investigations,  an  important  task  is  to  distinguish  laser  pulses  that  return  from  the 
ground  from  those  that  return  from  vegetation,  buildings,  or  other  objects.  This  task  is  carried 
out  through  the  use  of  automated  algorithms  followed  by  inspection  and  editing  by  skilled 
operators. 

This  problem  is  most  difficult  in  the  case  of  lidar  points  close  to  the  ground  surface. 

Identification  of  returns  from  trees  is  relatively  straightforward,  but  low  brush  and  grass  are  more 
difficult,  and  results  depend  more  on  operator  experience  and  judgment.  Orthophoto  images 
provide  an  important  cross-verification  resource  for  the  operator  in  these  cases. 

At  the  lower  detection  limit  of  the  technology,  it  can  become  very  difficult  to  distinguish  laser 
returns  from  an  uneven  ground  surface  from  those  from  low  brush  or  grass.  The  problem  of 
point  classification  on  uneven  ground  surfaces  has  been  observed  for  many  years,  as  shown  in 
Figure  1. 
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Figure  1 :  Lidar  Point  Classification  on  Forest  Roads 


Lidar  data  collected  in  2001  in  a  forest  and  timber  harvest  area  in  southeast  Alaska. 
Green  points,  classified  as  non-ground  returns,  can  be  seen  on  the  surface  of  the  forest 
road  where  the  photo  shows  no  vegetation.  It  is  difficult  to  determine  whether  these 
“non-ground”  returns  are  artifacts  of  the  classification  process  or  result  from  the  uneven 
surface  of  the  dirt  road. 
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2.2.2  Overview  of  Potential  Classification  Approaches 

Prior  to  the  early  2000s,  vendors  typically  developed  proprietary  custom  software  for  point 
classification,  based  on  a  variety  of  different  approaches.  These  approaches  varied  in  their 
success,  and  the  development  of  faster  laser  sensors  that  produced  higher  data  densities 
occasionally  led  to  unexpected  negative  results,  as  shown  in  Figure  2. 

Figure  2:  Early  Lidar  Point  Classification  Errors,  Alaska 
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Lidar  data  collected  in  2001  in  a  southeast  Alaska  forest.  This  was  the 
first  data  collection  flight  by  a  new  20kHz  sensor  system,  considered  very 
high-speed  at  the  time,  which  had  replaced  a  4kHz  system.  The  DEM 
shows  the  output  of  the  automated  classification  algorithm  developed  for 
the  earlier,  slower  system.  The  DEM  shows  numerous  “spikes”  caused 
by  reflections  from  trees  not  removed  by  the  automated  algorithm. 


Because  lidar  vendors  developed  proprietary  classification  algorithms,  there  is  little  published 
literature  available  about  the  methods  used.  The  small  amount  of  published  material  outlines 
the  following  general  approaches: 

■  Block-minimum  algorithm.  In  theory,  the  ground  returns  should  be  the  lowest 
points  in  a  lidar  data  set.  This  observation  suggests  the  use  of  a  block-minimum 
function  to  separate  ground  from  non-ground  returns.  The  algorithm  would 
establish  a  block  size  and  separate  the  lowest  points  in  that  block  as  potential 
ground  returns.  The  algorithm  could  be  adjusted  along  various  parameters  such 
as  block  size  and  establishment  of  specified  elevations  above  the  initial  surface 
within  which  points  would  be  classified  as  ground  returns.  Implementation  of 
block  minimum  functions  have  been  described  by  Kilan  et  al.  (1996),  and  Hansen 
and  Vogtle  (1999).  Haugerud  and  Harding  (2001)  observed  that  proprietary 
algorithms  then  in  use  by  some  North  American  lidar  vendors  appeared  to  be 
block-minimum  algorithms. 

■  Iterative  Differential  algorithm.  Anderson,  Reutebuch  and  McGaughey 
(Anderson,  et.al.  2006)  modified  a  method  described  by  Kraus  and  Pfeifer 
(1998).  This  is  an  iterative  method  that  begins  by  computing  an  initial  surface 
model  using  the  average  elevation  of  all  returns  within  a  1  m  x  1  m  grid  cell.  This 
initial  surface  model  is  computed  using  all  returns  from  both  vegetation  and 
ground.  For  each  iteration,  difference  values  are  computed  as  the  difference 
between  the  return  elevation  and  the  modeled  elevation  at  that  point.  Ground 
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returns  are  more  likely  to  be  below  the  modeled  surface,  and  thus  have  negative 
differences.  Vegetation  returns  are  more  likely  to  have  small  negative  or 
positive  differences.  These  residual  values  are  used  to  compute  weights  for 
each  return  using  a  weight  function.  Cells  with  no  ground  returns  are  flagged  as 
holes  in  the  intermediate  surface  and  filled  through  interpolation  from  surrounding 
cells.  Each  iteration  results  in  a  new  intermediate  surface  model.  The  authors 
reported  that  five  iterations  were  sufficient  to  remove  vegetation  while  preserving 
laser  returns  that  defined  ground  features,  while  additional  iterations  began  to 
remove  ground  features.  These  authors  performed  an  additional  final  step  of 
adding  all  points  within  15  centimeters  (cm)  of  the  intermediate  surface  to  the 
values  used  to  interpolate  the  final  ground  surface  model.  This  method  was  also 
described  by  Kraus  and  Pfeifer  (2001). 

■  No-multiple  return  algorithm.  The  absence  of  multiple  returns  suggests  that 
the  laser  signal  has  reached  the  ground  surface,  at  least  locally.  A  potential 
approach  would  be  to  require  that  there  be  no  multiple  returns  within  a  specified 
contiguous  area.  Haugerud  and  Harding  (2001 )  tested  an  algorithm  of  this  type 
and  found  that  it  rarely  misidentified  bare-earth  reflections  as  tree  canopy  and  did 
not  falsely  reject  points  within  large  bare-earth  areas.  However,  this  algorithm 
does  not  provide  information  about  the  ground  surface  beneath  the  tree  canopy, 
since  there  will  seldom  be  a  large  enough  contiguous  area  with  no  multiple 
returns  to  give  meaningful  results. 

■  De-spike  algorithm.  A  de-spike  algorithm  proceeds  from  the  observation  that  a 
ground  surface  should  be  relatively  smooth  compared  to  non-ground  objects 
such  as  trees  or  buildings.  The  algorithm  searches  for  points  that  define  local 
areas  of  strong  ground  curvature,  and  removes  them  iteratively  until  a  smooth 
ground  surface  is  left.  Haugerud  and  Harding  (2001)  tested  a  de-spike  algorithm 
using  data  from  a  large  lidar  survey  in  Washington  state,  and  this  algorithm  was 
later  adopted  by  the  lidar  vendor  engaged  in  the  survey.  The  de-spike  algorithm 
was  found  to  create  surfaces  that  look  realistic  and  to  match  reality  where 
surveyed  ground  control  existed.  The  algorithm  retained  a  larger  number  of 
points  than  the  block-minimum  algorithms  tested,  and  appeared  to  retain  more 
points  than  commercial  block-minimum  algorithms.  The  algorithm  removed  small 
buildings  and  most  bridges,  but  did  not  remove  large,  low  buildings  which  created 
large  flat  surfaces.  Haugerud  and  Harding  reported  several  disadvantages  of  the 
de-spike  algorithm,  including  long  computation  times1,  removal  of  some  points  at 
the  intersection  of  surfaces  such  as  the  toe  of  highway  cuts,  and  sensitivity  to 
negative  blunders  in  the  lidar  data.2 


1  Possibly  due  to  its  implementation  in  ArcGIS. 

2  Haugerud  and  Harding  observed  (as  have  others)  that  many  lidar  surveys  contain  a  few  points  that  are 
dramatically  lower  than  their  surroundings  and  clearly  not  reflected  from  actual  ground  features.  If  these 
blunders  are  not  removed,  they  appear  in  the  surface  model  as  conical  pits  in  the  ground  surface. 
Positive  blunders  are  also  observed,  possibly  resulting  from  laser  returns  from  birds  or  aerosols  in  the 
atmosphere.  These  appear  as  very  tall  “spikes”  and  would  be  removed  by  a  de-spike  algorithm. 
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2.2.3  Current  Classification  Methods  and  Software 

Beginning  in  the  early  2000s,  most  vendors  began  to  use  the  same  suite  of  software 
applications  for  processing  lidar  data,  including  point  classification.  This  set  of  programs  is 
produced  by  the  Finnish  company  TerraSolid3,  and  is  implemented  as  an  add-on  to  Bentley 
Microstation,  a  commonly-used  CAD  program.  Some  of  the  more  important  modules  in  the 
TerraSolid  software  suite  include: 

■  TerraScan:  Used  to  import  the  sensor  output,  create  data  points,  and  classify 
data  points  (Soininen  2005).  The  TerraScan  Viewer  is  also  available,  with  a 
more  limited  set  of  tools  for  viewing  and  modeling  lidar  points. 

■  TerraModeler:  Used  to  validate  laser  points  and  create  surface  models  that  are 
used  to  create  contours,  elevation  models,  profiles  and  cross  sections,  volumes, 
etc. 

■  TerraMatch:  Used  to  resolve  survey  parameters  and  correct  for  roll,  pitch, 
heading,  and  elevation  changes  between  flight  lines. 

■  TerraSurvey:  Used  to  import  survey  data,  display  profiles,  and  perform  data 
quality  control  (QC)  functions. 

■  TerraPhoto:  Used  to  process  and  rectify  digital  images  to  create  orthophotos, 
including  using  lidar  data  as  part  of  the  rectification  process. 

TerraScan,  the  module  used  for  point  classification,  uses  a  sophisticated  de-spike  approach, 
with  a  variety  of  adjustable  parameters.  The  initial  classification  takes  place  in  two  steps: 

■  Selecting  low  points.  This  step  creates  a  temporary  ground  surface  using  the 
lowest  points  in  the  data  set.  Each  point  in  the  data  set  is  examined,  and  its 
elevation  is  compared  to  all  other  points  within  a  specified  horizontal  distance. 
The  lowest  point  is  classified  as  ground.  The  routine  can  also  search  for  groups 
of  low  points  rather  than  single  points.  This  step  includes  routines  to  eliminate 
isolated  negative  or  positive  blunders4. 

■  Adding  points  to  the  ground  surface  model.  This  step  classifies  ground 
returns  by  iteratively  adding  points  to  the  initial  temporary  ground  surface.  The 
routine  starts  by  using  the  low  points  selected  in  the  first  step  to  create  a 
triangulated  surface  model,  with  the  vertices  of  the  triangles  at  the  modeled 
ground  surface.  Because  these  are  the  lowest  points  in  the  data  set,  the 
remaining  points  are  all  higher  than  this  initial  ground  surface.  The  model 
iteratively  adds  a  new  laser  point  to  the  triangulated  model. 

Four  parameters  influence  the  results  of  point  classification: 

■  Maximum  building  size.  This  parameter  sets  the  largest  area  in  which  the 
program  will  select  an  initial  low  point.  If  the  setting  is  too  small,  low  points  will 
be  established  on  large  flat  surfaces  such  as  building  roofs.  If  the  setting  is  too 


3  See:  http://www.terrasolid.fi/en/products 

4  At  approximately  the  time  TerraScan  became  the  most  popular  lidar  processing  software,  negative  and 
positive  blunders  largely  stopped  appearing  in  the  lidar  data  delivered  from  vendors. 
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large,  natural  features  such  as  small  hills,  stream  channels,  or  ditches  will  be 
missed. 

■  Iteration  Angle.  This  parameter  sets  the  maximum  angle  from  the  nearest  point 
already  in  the  ground  surface  model.  A  low  value  is  more  appropriate  for  flatter 
sites  with  gradually  changing  topography,  while  a  higher  maximum  angle  would 
be  appropriate  for  steeper  or  more  rugged  terrain. 

■  Iteration  distance.  This  parameter  sets  the  maximum  value  for  the  height  of  a 
potential  point  above  the  existing  ground  surface  model.  As  with  iteration  angle, 
low  values  are  more  appropriate  for  flatter  sites  and  higher  values  are  more 
appropriate  for  steeper  sites. 

■  Terrain  angle.  Defines  the  “steepest  allowed  slope  in  ground  terrain”.  The 
terrain  angle  is  especially  important  in  the  “first-cut”  or  the  first  routine  used  to 
delineate  the  ground  model,  which  all  other  ground  classification  routines  will  use 
as  their  reference.  If  this  value  is  set  too  low,  subsequent  routines  will  stand  little 
chance  of  correctly  classifying  ground  in  steep  terrain.  If  set  too  high,  vegetation 
may  erroneously  be  classified  as  ground.  The  terrain  angle  should  be  selected 
to  match  the  unique  conditions  present  in  each  terrain  model. 

Final  editing  by  the  operator  is  used  to  correct  for  errors  that  remain  following  the  automated 
process.  The  overall  process  is  shown  in  Figure  3. 


Figure  3:  Lidar  Point  Classification  Process 


13 


ESTCP  White  Paper:  Effects  of  LiDAR  Point  Classification  Methods 
on  Surface  Model  Creation  and  Feature  Identification 
Project  Number  07  E-MM2-012/MM-0737 


Figure  3:  Lidar  Point  Classification  Process  (continued) 


The  first  classification  steps  in 
TerraSolid  are  to: 

•  Establish  maximum 
building  size  (blue  lines). 

•  Identify  initial  low  points 
(red  points)  for  each  area. 

•  Using  the  initial  low  points, 
create  the  initial  ground 
surface  model. 


The  next  step  is  to  evaluate 
each  remaining  point  for 
addition. 

Evaluation  is  based  on  the 
elevation  above  the  existing 
surface  model  and  the  angle 
with  the  closest  already- 
accepted  point. 

In  this  sketch,  the  point  on  the 
left  would  be  classified  as 
ground  and  the  point  on  the 
right  as  non-ground. 


The  final  ground 
classifications  are  used  to 
create  the  DEM  representing 
the  modeled  ground  surface. 
When  done  well,  the  modeled 
surface  closely  corresponds 
to  the  actual  ground 
conditions. 


TerraSolid’s  automated  algorithms  will  classify  points  with  varying  degrees  of  success, 
depending  on  the  fit  between  the  parameter  settings  and  the  terrain  type  being  surveyed.  No 
one  combination  of  parameter  settings  will  be  optimum  for  all  sites  or  even  all  parts  of  the  same 
site.  Frequently,  the  operator  will  divide  the  site  into  a  series  of  terrain  types,  each  with 
somewhat  different  settings.  The  labor  tradeoff  between  subsequent  hand  editing  and 
establishing  additional  terrain  types  is  a  function  of  operator  experience.  As  many  clients  prefer 
to  see  a  “smooth”  ground  surface,  most  vendors  by  default  tailor  the  parameter  settings  for  each 
terrain  type  so  that  the  resulting  ground  is  a  smooth  as  possible  while  still  conforming  to  the 
project’s  accuracy  and  density  specifications. 

Several  approaches  would  permit  larger  numbers  of  points  to  be  classified  as  ground,  including 
establishing  different  iteration  elevations  or  angles.  TerraSolid  also  will  permit  bulk  classification 
of  all  points  within  a  given  elevation  of  the  initial  surface  as  ground,  regardless  of  iteration  angle. 
All  of  these  approaches  would  result  in  a  more  detailed  ground  surface,  although  such 
approaches  could  also  include  additional  roughness  from  system  noise,  variations  in  the  ground 
surface,  and  low  vegetation. 
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A  classification  scheme  that  resulted  in  more  ground  points  could  be  produced  either  in  place  of 
or  in  addition  to  the  vendor’s  initial  classification  approach.  Establishing  an  additional 
classification  cateogory  (“near  ground”  in  addition  to  “ground”  “non-ground”)  would  allow 
Government  land  managers  to  use  the  initial  classifications  in  those  applications  where  a 
smoother  ground  surface  was  desirable,  such  as  for  creating  contour  lines,  while  allowing  for 
creation  of  more  detailed  surface  models  for  studies  of  potential  munitions-related  features, 
where  the  addition  of  some  additional  surface  roughness  could  be  acceptable. 
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3  Point  Classification  Results  at  Six  Sites 

This  section  describes  the  results  of  lidar  point  classification  at  the  four  ESTCP  sites  where  lidar 
data  was  collected,  along  with  two  non-ESTCP  sites.  Since  more  than  one  lidar  flight  was 
conducted  at  some  ESTCP  sites,  a  total  of  1 1  lidar  data  sets  were  examined.  The  objectives  of 
this  section  are  to  determine  the  percentage  of  lidar  points  that  were  classified  as  non-ground 
returns,  and  the  processes  that  account  for  the  classification  results. 

3.1  Results  -  Former  Camp  Beale  Demonstration  Site 

The  Former  Camp  Beale  site  is  located  near  Marysville,  California.  Data  was  acquired  as  part 
of  the  ESTCP  WAA  Pilot  Program  in  July  2006.  Lidar  was  collected  and  processed  by  Terra 
Remote  Sensing,  Inc. 

3.1.1  Lidar  Point  Patterns 

Lidar  is  collected  in  a  series  of  overlapping  flight  lines.  The  amount  of  flight  line  overlap  is 
determined  by  project  requirements  and  flight  conditions.  All  of  the  three  vendors  contacted 
plan  for  a  minimum  overlap  of  15  to  50  percent,  but  in  practice,  overlap  may  vary  considerably 
during  the  survey  based  on  flight  conditions  (Figure  4). 


Figure  4:  Flight  Line  Overlap 


Example  area.  Images  show  orthophoto,  digital  terrain  model  (DTM)  created  from  all  lidar  points,  and  DEM  created 
from  ground  points  only. 


Lidar  points  from  overlapping  flight  lines.  The  aircraft  traveled  east  -  west.  The  impact  of  wind  conditions  can  be 
seen  at  the  edges  of  the  flight  lines. _ 
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As  the  mirror  sweeps  back  and  forth,  lidar  points  fall  in  roughly  parallel  lines,  perpendicular  to 
the  direction  of  the  flight  line  (Figure  5). 


Figure  5:  Lidar  Point  Patterns 


Test  area  showing  lines  of  lidar  points  perpendicular  to  the  direction  of  the  flight  line.  The  flight  lines  reach  the 
ground  in  a  long  “z”  pattern  as  the  sensor  sweeps  back  and  forth  and  the  aircraft  moves  forward;  however,  since 
the  lines  are  long  compared  to  the  angle,  they  appear  as  roughly  parallel. 


Objects  such  as  trees  change  the  patterns  of  lidar  points,  and  such  objects  can  create  small 
“shadowed”  areas.  These  areas  are  caused  by  the  angle  of  approach  of  the  lidar  points  toward 
the  ground  surface.  The  effect  is  clearly  seen  in  areas  of  flight  line  overlap  where  the  “shadow” 
effect  is  different  for  each  flight  line  (Figure  6).  These  shadowed  areas  can  generally  be 
eliminated  through  overlapping  flight  lines. 

3.1 .2  Lidar  Data  Density  Variations 

All  lidar  points.  Lidar  point  density  varies  with  flight  line  overlap  and  flight  conditions  (URS 
2007a).  For  the  Former  Camp  Beale  site,  point  density  variations  were  illustrated  by  creating 
density  grids  in  which  each  1  m  cell  was  labeled  with  the  number  of  lidar  points  in  the  cell. 
These  grids  clearly  show  the  higher  point  density  in  areas  of  flight  line  overlap,  and  also  show 
higher  data  density  where  the  aircraft  pitches  into  the  wind.  These  appear  as  bands  of  higher 
density  perpendicular  to  the  flight  line  direction,  often  giving  the  appearance  of  “curtain  folds” 
(Figure  7). 
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Figure  6:  Effects  of  Vegetation  on  Lidar  Point  Patterns 


Effects  of  trees  on  lidar  point  patterns.  Orthophoto  and  all-points  DTM. 


Changes  in  lidar  point  patterns,  overlapping  flight  lines.  The  tree  in  the  image  on  the  left  is  near  the  north  extent  of 
the  flight  line,  the  same  tree  in  the  image  to  the  right  is  near  the  center  of  the  flight  line.  Small  shadowed  areas 
can  be  seen  where  no  lidar  points  reach  the  ground,  and  these  shadowed  areas  are  different  for  each  flight  line.. 
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Figure  7:  Lidar  Data  Density  Grids 


Data  density  grids,  same  data 
block  as  Figure  4. 

Darker  colors  indicate  higher 
data  density.  Areas  of  higher 
density  correspond  to  areas  of 
flight  line  overlap  and  also 
appear  where  the  aircraft 
pitched  and  rolled,  creating 
“curtain”  effects. 

The  overall  point  density  for 
the  Former  Camp  Beale  site 
was  13.8  pts/m2. 

Values  for  this  block: 

•  Maximum:  126  pts/m2 

•  Minimum:  0  pts/m2 

•  Mean:  12.42  pts/m2 

•  Standard  deviation:  6.47 

pts/m2 


The  histogram  shows  that 
while  the  maximum  value  is 
126  pts/m2,  the  highest  values 
are  outliers.  The  majority  of 
the  values  are  between  0  and 
31  pts/m2,  with  the  most 
common  values  near  the 
overall  average  density  of  13.8 
pts/m2. 


Point  density  variations  are  further  illustrated  in  Figure  8. 


19 


ESTCP  White  Paper:  Effects  of  LiDAR  Point  Classification  Methods 
on  Surface  Model  Creation  and  Feature  Identification 
Project  Number  07  E-MM2-012/MM-0737 


Figure  8:  Lidar  Point  Density  Variation 


Point  density  test  area,  orthophoto  and  lidar  points. 


Point  density  images.  The  image  on  the  left  shows  a  color  ramp  between  minimum  and  maximum  values;  the 
image  on  the  right  shows  the  same  data  classified  in  bands  of:  0-4,  5-10,  1 1-25,  and  26-126  pts/m2.  The  image 
on  the  right  shows  that  in  the  non-overlap  area  (upper  right),  most  cells  have  between  5  and  10  pts/m2,  and  in  the 
flight  line  overlap  area  (lower  left)  most  cells  have  between  1 1  and  25  pts/m2.  The  area  of  aircraft  pitch  (the  center 
“stripe”)  often  has  over  25  pts/m2 


Ground  returns .  The  density  of  ground  returns  can  be  mapped  in  a  manner  similar  to  the  all- 
points  maps  shown.  Figure  9  shows  the  same  area  as  Figures  7  using  only  returns  classified  by 
the  vendor  as  ground.  In  Figure  9,  the  contrast  between  all-points  density  and  ground-points 
density  is  dramatic.  The  maximum  point  density  was  decreased  from  126  total  points  to  8 
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ground  pts/m2.  Areas  of  extremely  high  point  density  have  been  eliminated.  The  mean  value 
for  all  points  in  the  test  block  is  12.42  pts/m2,  which  is  roughly  comparable  to  the  value  for  the 
entire  demonstration  site  of  13.8  pts/m2.  The  overall  density  of  ground  points  is  much  lower,  at 
1 .78  pts/m2.  Variability  is  also  lower  for  the  ground  points,  as  illustrated  by  the  fact  that  the 
ground  point  density  map  is  relatively  uniformly  colored  while  the  all-points  image  highly 
variable. 

Tthe  density  of  the  vendor-classified  ground  returns  did  not  show  higher  density  in  areas  of  flight 
line  overlap  or  aircraft  pitch  and  roll.  This  lower  variability  of  the  ground  points  is  initially 
puzzling  since  ground  points  should,  intuitively,  be  more  dense  in  areas  of  flight  line  overlap  and 
should  be  higher  in  areas  of  aircraft  pitch  in  the  same  way  as  overall  point  density. 


Figure  9:  Lidar  Data  Density  -  All  Points  and  Ground  Points 


All  points: 

Minimum:  0  pts/m2 

Vendor-classified  ground  points  only: 

Minimum:  0  pts/m2 

Maximum:  126  pts/m2 

Maximum:  8  pts/m2 

Mean:  12.42  pts/m2 

Mean:  1.78  pts/m2 

Standard  deviation:  6.47 

Standard  deviation:  1.07 
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Figure  9:  Lidar  Data  Density  -  All  Points  and  Ground  Points  (continued) 


Point  density  test  area,  repeated  from  Figure  8.  The  image  on  the  right  shows  density  for  the  full  lidar  data  set. 
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Density  for  the  ground  points  only,  same  area.  The  image  on  the  right  shows  the  vendor’s  ground  points  (black) 
and  non-ground  points  (green). 


3.1.3  Point  Classification 


Initial  observations.  At  the  Former  Camp  Beale  site,  many  lidar  points  appeared  to  have  been 
classified  as  “non-ground”  returns,  even  in  areas  where  there  is  little  or  no  apparent  vegetation. 
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Figure  10  shows  an  area  of  paved  road  surface  with  the  lidar  points  for  the  same  area  color 
coded  by  ground  or  non-ground  returns. 


Figure  10:  Lidar  Classification  Results  on  Paved  Road  Surfaces 


Orthophoto  and  classified  lidar  points  for  the  initial  point  classification  test  area.  In  the  image  on  the  right,  the 
vendor’s  ground  points  are  black,  and  the  vendor’s  non-ground  points  are  green.  The  road,  which  should  not 
have  any  non-ground  returns,  is  not  visible. 


This  image  shows  that  the  road — where  there  are  no  true  “non-ground”  returns — has  many 
returns  classified  by  the  vendor  as  non-ground.  Further,  the  classification  for  the  paved  surface 
appears  to  be  similar  to  the  sparsely-vegetated  areas  on  either  side  of  the  road.  In  both  areas, 
it  appears  that  a  large  number  of  points  classified  as  non-ground  could  have  been  classified  as 
ground  returns. 

The  points  classified  as  non-ground  are  generally  not  used  in  the  creation  of  lidar-based  surface 
models.  However,  these  points  were  not  “lost”,  since  the  vendor  provided  the  full  lidar  data  set. 

Test  areas.  In  response  to  these  initial  findings,  a  series  of  14  test  areas  was  chosen,  all  on 
paved  roads  where  site  photos  showed  no  overhanging  vegetation.  In  these  areas,  all  “non¬ 
ground”  lidar  returns  were  assumed  to  be  artifacts  of  the  classification  process.  For  each  site, 
the  number  of  ground  and  non-ground  returns  was  measured,  and  point  density  calculated  for 
all  points  and  “ground”  points.  Test  areas  were  chosen  in  both  areas  of  flight  line  overlap  and 
non-overlap.  The  results  are  shown  in  Table  1. 
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Table  1 :  Former  Camp  Beale  Test  Area  Results 


Area  ID 

Area 

(m2) 

Number  of  Points 

Percentage 

Density  (pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non- 

Ground 

All 

Ground 

1 

516 

1,399 

715 

684 

51.1 

48.9 

2.71 

1.39 

2 

398 

1,276 

606 

670 

47.5 

52.5 

3.21 

1.52 

3 

2,200 

7,815 

3,494 

4,321 

44.7 

55.3 

3.55 

1.59 

4 

492 

1,913 

704 

1,209 

36.8 

63.2 

3.89 

1.43 

5 

871 

6,614 

1,887 

4,727 

28.5 

71.5 

7.59 

2.17 

6 

524 

4,966 

916 

4,050 

18.4 

81.6 

9.48 

1.75 

7 

763 

7,732 

1,453 

6,279 

18.8 

81.2 

10.13 

1.90 

8 

784 

8,098 

1,358 

6,740 

16.8 

83.2 

10.33 

1.73 

9 

2,477 

29,463 

4,908 

24,555 

16.7 

83.3 

11.89 

1.98 

10 

1,450 

20,440 

2,729 

17,711 

13.4 

86.6 

14.10 

1.88 

11 

662 

9,902 

1,333 

8,569 

13.5 

86.5 

14.96 

2.01 

12 

539 

9,340 

2,256 

7,084 

24.2 

75.8 

17.33 

4.19 

13 

2,118 

40,297 

4,046 

36,251 

10.0 

90.0 

19.03 

1.91 

14a 

431 

17,888 

828 

17,065 

4.6 

95.4 

41.50 

1.92 

a.  This  test  area  included  both  fight  line  overlap  and  aircraft  pitch,  which  resulted  in  a  very  high  overall  point  density. 


One  result  of  this  test  was  that  the  percentage  of  laser  returns  classified  as  non-ground 
increased  closely  with  the  increase  in  overall  density.  A  single  “background”  rate  of  non-ground 
classification  cannot  be  developed,  since  the  rate  of  non-ground  classification  varies  with  overall 
point  density. 

A  second  result  is  that  as  the  overall  data  density  rose  from  2.7  to  41 .5  pts/m2,  ground  point 
density  rose  much  more  modestly,  from  around  1 .4  to  around  2.2  pts/m2  (with  only  one  outlier). 

3.1.4  Vertical  Distribution  of  Non-Ground  Points 

Initial  observations.  Most  of  the  points  classified  as  “non-ground”  appear  to  be  close  to  the 
ground  surface.  As  an  initial  test,  lidar  points  for  the  same  area  were  re-classified  to  include  all 
of  the  points  that  are  within  0.4  m  of  the  ground  surface  in  the  “ground”  classification.  Figure  1 1 
shows  that  this  approach  added  almost  all  of  the  points  to  the  ground  surface  except  those 
reflecting  from  trees  or  higher  ground  features. 
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Figure  11:  Initial  Point  Reclassification  Test  -  Former  Camp  Beale 


Initial  point  classification  test  area,  points  re-classified.  “Non-ground”  points  within  0.4m  of  the  modeled  ground 
surface  were  added  to  the  “ground”  classification.  The  remaining  “non-ground”  points  appear  to  be  reflections  from 
trees  and  other  objects  on  the  ground.  The  image  to  the  right  is  the  DTM  using  the  full  lidar  data  set. 


Test  areas.  The  vertical  distribution  of  the  “non-ground”  returns  was  examined  for  six  of  the 
point  density  test  areas.  The  areas  examined  were  all  paved  surfaces  with  no  apparent 
overhanging  vegetation  or  unpaved  road  shoulder.  Non-ground  returns  were  measured  for  their 
distance  above  or  below  the  surface  model  created  from  the  points  classified  as  ground  returns. 
Four  elevation  classes  were  established:  +/-  5  cm,  between  +/-  5  and  10  cm,  between  +/- 10 
and  15  cm,  and  greater  than  15  cm.  For  each  of  the  test  areas,  the  majority  of  the  “non-ground” 
returns  were  within  10  cm  of  the  modeled  surface  (Table  2).  Only  a  small  number  (between  3.1 
and  15.3  percent)  were  more  than  15  cm  from  the  modeled  surface. 


Table  2:  Elevation  Distribution  for  Non-Ground  Points  -  Former  Camp  Beale 


Test 
Area  ID 

Area 

(m2) 

Number  of 
Points  - 
Non-Ground 

Elevation  Class  (%) 

<=  +/-  0.05 

cm 

>  0.05  and 
<=  0.10  cm 

>  0.10  and 
<=  0.15  cm 

>0.15 

cm 

1 

398 

670 

58.8 

30.4 

8.4 

2.4 

2 

763 

6,279 

35.4 

37.0 

20.8 

6.8 

4 

539 

7,084 

24.3 

36.4 

24.0 

15.3 

7 

492 

1,209 

53.8 

32.6 

10.4 

3.1 

8 

662 

8,569 

29.2 

36.0 

23.0 

11.8 

10 

1,450 

17,711 

27.3 

35.4 

25.1 

12.3 

The  elevation  distribution  of  the  non-ground  returns  is  illustrated  in  Figure  12.  The  general 
elevation  distribution  of  non-ground  points  appears  to  extend  to  the  relatively  un-vegetated  area 
off  the  road  surface. 
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Figure  12:  Elevation  Distribution  for  “Non-Ground”  Points-  Former  Camp  Beale 


Vertical  distribution  of  non-ground  returns.  There  were  no 
points  further  than  10  cm  from  the  modeled  ground  surface 
in  this  area. 


Vertical  distribution  of  non-ground  returns,  on  and  off  paved 
surface.  There  were  no  points  further  than  10  cm  from  the 
modeled  ground  surface  in  this  area. 


These  results  suggest  that  at  least  the  majority  of  the  points  classified  as  non-ground  could  be 
added  back  to  the  ground  surface  model  with  little  risk  of  including  low  vegetation  or  other  “true” 
non-ground  features. 

3.2  Results  at  Additional  Sites 

Lidar  data  from  six  additional  sites  was  examined  to  determine  whether  the  same  types  of 
effects  would  be  found.  Sites  included: 

■  Kirtland  Air  Force  Base  (AFB)  Precision  Bombing  Range  (PBR).  Data 
collected  by  Terra  Remote  Sensing  in  August  2005.  Four  lidar  data  sets  were 
collected. 

■  Victorville  Demolition  Bombing  Target  (DBT)  “Y”.  Data  collected  by  Terra 
Remote  Sensing  in  January  2006.  Two  lidar  data  sets  were  collected. 

■  Pueblo  PBR.  Data  collected  by  Sky  Research  in  August  2004  (Phase  I)  and 
August  2005  (Phase  II).  One  lidar  data  set  was  collected. 

■  Former  Camp  Beale  Sky  Research.  In  May  2006,  Sky  Research  collected  lidar 
data  for  Beale  AFB.  Data  collection  involved  some  overlap  with  the  Former 
Camp  Beale  demonstration  site.  One  lidar  data  set  was  collected. 

■  Puget  Sound  Lidar  Coalition,  Portland  Area.  Data  collected  by  Terrapoint  in 
2004.  One  lidar  data  set  was  collected. 
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The  two  Puget  Sound  sites  were  collected  at  a  lower  data  density,  since  their  primary  purpose 
was  not  to  detect  small  surface  features.  They  are  included  to  represent  the  work  of  an 
additional  lidar  vendor  and  a  lower  point  density.  The  sites  selected  are  not  intended  to  be  a 
comprehensive  survey  of  lidar  data  sets,  but  rather  to  be  an  overview  to  determine  whether 
results  appear  similar  to  those  at  the  ESTCP  sites.  Results  are  described  in  the  sections  below. 

3.2.1  Kirtland  Air  Force  Base  Precision  Bombing  Range 

Four  lidar  flights  were  conducted  at  this  site,  as  part  of  an  investigation  of  point  density  required 
to  detect  features. 

Point  density  maps.  Figure  13  shows  point  density  for  the  full  point  data  set  and  for  ground 
points  alone  for  one  of  the  300  m  flights  conducted  at  this  site.  Results  for  all  four  flights  were 
similar  and  are  discussed  in  Appendix  C  to  the  ESTCP  Final  Report  for  this  demonstration  site 
(URS  2007a). 


Test  areas.  Tables  3  through  6  show  the  results  for  10  test  areas,  each  50  m  x  50  m.  These 
areas  were  selected  to  represent  both  flightline  overlap  and  non  overlap  areas  for  all  four  of  the 
lidar  flights  conducted.  The  Kirtland  site  did  not  contain  any  paved  roads,  so  it  was  not  possible 
to  select  test  areas  where  non-ground  classification  would  be  assured  to  be  classification 
artifacts.  However,  vegetation  conditions  did  not  vary  noticeably  over  the  site,  so  in  theory  the 
percentage  of  vegetation  returns  should  be  consistent  for  all  test  areas  and  for  all  lidar  flights. 
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Table  3:  Kirtland  AFB  PBR  Test  Area  Results 
900  m  Flight 


Area  ID 

Area 

(m2) 

1 

Slumber  ol 

Points 

Percentage 

Density  (pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non- 

Ground 

All 

Ground 

1 

2,500 

4,222 

2,970 

1252 

70.35 

29.65 

1.69 

1.19 

2 

2,500 

4,566 

2,929 

1637 

64.15 

35.85 

1.83 

1.17 

3 

1,908 

60.51 

39.49 

1.93 

1.17 

4 

m 

1,526 

68.46 

■■ 

1.94 

1.33 

5 

EKH 

1,531 

68.67 

mum 

1.95 

1.34 

6 

1,532 

68.87 

31.13 

1.97 

1.36 

7 

2,613 

57.79 

42.21 

2.48 

1.43 

8 

2,491 

61.03 

38.97 

2.56 

1.56 

9 

2,500 

2,571 

60.28 

39.72 

2.59 

1.56 

10 

2,500 

3,572 

46.35 

53.65 

2.66 

1.23 

Table  4:  Kirtland  AFB  PBR  Test  Area  Results 
450  m  Flight 


Area  ID 

■HI 

N 

umber  of 

Points 

Percentage 

Density(pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non- 

Ground 

All 

Ground 

1 

7,430 

4,342 

3088 

58.44 

41.56 

2.97 

1.74 

2 

2,500 

9,569 

4619 

51.73 

48.27 

3.83 

1.98 

3 

2,500 

10,367 

48.01 

51.99 

4.15 

1.99 

4 

5,392 

6,869 

43.98 

56.02 

4.90 

2.16 

5 

5,827 

29.86 

70.14 

7.81 

2.33 

6 

6,451 

29.74 

70.26 

8.68 

2.58 

7 

2,500 

24.91 

75.09 

8.95 

2.23 

8 

2,500 

25.06 

74.94 

10.14 

2.54 

9 

2,500 

22.43 

77.57 

10.98 

2.46 

10 

2,500 

34,190 

6,461 

27,729 

18.90 

81.10 

13.68 

2.58 
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Table  5:  Kirtland  AFB  PBR  Test  Area  Results 
300  m  Flight  East-West  Flight  Lines 


Area  ID 

■HI 

Number  of  Points 

Percentage 

Density 

(pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non-Ground 

All 

Ground 

1 

8,199 

4,347 

3852 

53.02 

46.98 

3.28 

1.74 

2 

2,500 

43.43 

56.57 

4.16 

1.81 

3 

2,500 

40.74 

59.26 

5.30 

2.16 

4 

9,989 

33.09 

66.91 

5.97 

1.98 

5 

35.36 

64.64 

6.19 

2.19 

6 

28.03 

71.97 

7.08 

1.98 

7 

15,390 

29.38 

70.62 

8.72 

2.56 

8 

18,038 

26.80 

73.20 

9.86 

2.64 

9 

18,569 

25.88 

74.12 

10.02 

2.59 

10 

2,500 

25,749 

6,778 

18,971 

26.32 

73.68 

10.30 

2.71 

Table  6:  Kirtland  AFB  PBR  Test  Area  Results 
300  m  Flight  North-South  Flight  Lines 


Area  ID 

Area 

(m2) 

Number  of  Points 

Percentage 

Density  (pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non- 

Ground 

All 

Ground 

1 

8306 

4,500 

3806 

54.18 

45.82 

3.32 

1.80 

2 

9052 

4,734 

4318 

52.30 

47.70 

3.62 

1.89 

3 

2,500 

9486 

4,884 

4,602 

51.49 

48.51 

3.79 

1.95 

4 

2,500 

10479 

5,475 

47.75 

52.25 

4.19 

2.00 

5 

11513 

5,321 

6,192 

46.22 

53.78 

4.61 

2.13 

6 

2,500 

11680 

6,273 

46.29 

53.71 

4.67 

2.16 

7 

2,500 

13855 

5,948 

7,907 

42.93 

57.07 

5.54 

2.38 

8 

2,500 

14954 

6,553 

8,401 

43.82 

56.18 

5.98 

2.62 

9 

2,500 

16055 

6,089 

9,966 

37.93 

62.07 

6.42 

2.44 

10 

2,500 

21650 

6,403 

15,247 

29.58 

70.42 

8.66 

2.56 

Comparison  with  previous  data  sets.  As  with  the  Former  Camp  Beale  data,  the  density  map 
for  the  full  data  set  clearly  shows  the  effects  of  flight  line  overlap,  and  this  effect  is  largely 
absent  in  the  density  map  based  on  the  vendor’s  ground  points  (Figure  13).  Ground  point 
density  is  much  less  variable  than  overall  density  (Tables  3  through  6).  As  overall  point  density 
went  up,  the  percentage  of  non-ground  classifications  also  went  up.  As  at  Former  Camp  Beale, 
the  ground  point  density  was  generally  no  higher  than  2.5  pts/m2,  even  when  overall  density 
was  as  high  as  13.68  pts/m2. 

3.2.2  Victorville  Demolition  Bombing  Target  “Y” 

Two  lidar  flights  were  conducted  at  this  site. 
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Point  density  maps.  Figure  14  shows  point  density  for  the  full  point  data  set  and  for  ground 
points  alone,  for  one  of  the  two  lidar  flights  conducted  at  this  site.  Results  for  both  flights  are 
similar  and  are  discussed  in  Appendix  C  to  the  ESTCP  Final  Report  for  this  demonstration  site 
(URS  2007a). 


Figure  14:  Point  Density  Maps  -  Victorville  DBT  “Y” 


Test  areas.  Tables  7  and  8  show  the  results  for  10  test  areas,  each  25  m  x  25  m.  As  with  the 
Kirtland  site,  the  Victorville  site  did  not  contain  any  paved  roads,  so  test  areas  were  chosen  in 
open  areas.  However,  the  site  does  not  contain  tall  vegetation,  and  vegetation  conditions  do 
not  vary  noticeably  over  the  site,  so  in  theory  the  percentage  of  vegetation  returns  should  be 
consistent  for  all  test  areas,  and  for  all  lidar  flights. 


Table  7:  Victorville  DBT  “Y”  Test  Area  Results 
300  m  Flight 


Area  ID 

Area 

(m2) 

Number  of  Points 

Percentage 

Density 

(pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non-Ground 

All 

Ground 

1 

625 

2,335 

1,441 

894 

61.71 

38.29 

3.74 

2.31 

2 

625 

2,826 

1,490 

1,336 

52.72 

47.28 

4.52 

2.38 

3 

625 

2,851 

1,274 

1,577 

44.69 

55.31 

4.56 

2.04 

4 

625 

2,964 

1,248 

1,716 

42.11 

57.89 

4.74 

2.00 

5 

625 

3,219 

1,420 

1,799 

44.11 

55.89 

5.15 

2.27 

6 

625 

3,518 

1,535 

1,983 

43.63 

56.37 

5.63 

2.46 

7 

625 

4,175 

1,410 

2,765 

33.77 

66.23 

6.68 

2.26 

8 

625 

5,059 

1,506 

3,553 

29.77 

70.23 

8.09 

2.41 

9 

625 

5,725 

1,829 

3,896 

31.95 

68.05 

9.16 

2.93 
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10 

625 

7,247 

1,859 

5,388 

25.65 

74.35 

11.60 

2.97 

Table  8:  Victorville  DBT  “Y”  Test  Area  Results 

450  m  Flight 

Area  ID 

Area 

(m2) 

Number  of  Points 

Percentage 

Density  (pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non- 

Ground 

All 

Ground 

1 

625 

1,789 

718 

59.87 

40.13 

2.86 

1.71 

2 

625 

1,910 

810 

57.59 

42.41 

3.06 

1.76 

3 

625 

2,085 

986 

52.71 

3.34 

1.76 

4 

625 

978 

58.29 

3.75 

1.56 

5 

625 

51.55 

3.93 

1.90 

6 

625 

3,614 

1,219 

2,395 

33.73 

66.27 

5.78 

1.95 

7 

625 

4,839 

na 

29.61 

70.39 

7.74 

2.29 

8 

625 

5,362 

■ESH 

27.06 

72.94 

8.58 

2.32 

9 

625 

4,010 

27.98 

72.02 

8.91 

2.49 

10 

625 

4,225 

24.57 

75.43 

8.96 

2.20 

Comparison  with  previous  data  sets.  As  with  the  Former  Camp  Beale  and  Kirtland  data,  the 
density  map  for  the  full  data  set  clearly  shows  the  effects  of  flight  line  overlap  and  aircraft 
motion,  which  is  largely  absent  in  the  density  map  based  on  the  vendor’s  ground  points.  As  with 
the  other  sites,  the  percentage  of  non-ground  returns  increased  roughly  with  overall  point 
density,  and  ground  point  density  is  no  higher  than  2.49  pts/m2,  even  when  overall  density  was 
as  high  as  8.96  pts/m2. 

3.2.3  Pueblo  Precision  Bombing  Range 

One  lidar  data  set  was  acquired  at  the  Pueblo  site. 

Point  density  maps.  Figure  15  shows  point  density  for  the  full  point  data  set  and  for  ground 
points  alone. 
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Figure  15:  Point  Density  Maps  -  Pueblo  PBR 


Test  areas.  Table  9  shows  the  results  for  15  test  areas,  each  just  over  50  x  50  m.  Like  the 
Kirtland  and  Victorville  sites,  the  Pueblo  site  did  not  contain  any  paved  roads,  so  it  was  not 
possible  to  select  test  areas  where  non-ground  classification  would  be  certain  to  be 
classification  artifacts.  However,  the  site  does  not  contain  high  vegetation,  and  vegetation 
conditions  do  not  vary  noticeably  over  the  site,  so  in  theory  the  percentage  of  vegetation  returns 
should  be  consistent  for  all  test  areas. 

Table  9:  Pueblo  PBR  Test  Area  Results 


Area  ID 

Area 

(m2) 

Number  of  Points 

Percentage 

Density  (pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non- 

Ground 

All 

Ground 

1 

2,500 

5,232 

4,580 

652 

87.54 

12.46 

2.09 

1.83 

2 

2,500 

5,626 

4,352 

1,274 

77.36 

22.64 

2.25 

1.74 

3 

2,500 

6,330 

5,036 

1,294 

79.56 

20.44 

2.53 

2.01 

4 

2,500 

8,496 

6,660 

1,836 

78.39 

21.61 

3.40 

2.66 

5 

2,500 

10,843 

5,794 

5,049 

53.44 

46.56 

4.34 

2.32 

6 

2,500 

1 1 ,728 

5,962 

5,766 

50.84 

49.16 

4.69 

2.38 

7 

2,500 

13,894 

7,481 

6,413 

53.84 

46.16 

5.56 

2.99 

8 

2,500 

14,117 

5,511 

8,606 

39.04 

60.96 

5.65 

2.20 

9 

2,500 

16,142 

7,858 

8,284 

48.68 

51.32 

6.46 

3.14 

10 

2,500 

16,283 

5,654 

10,629 

34.72 

65.28 

6.51 

2.26 

The  effect  of  flight  line  overlap  on  point  classification  can  be  seen  in  Figure  16,  which  shows  the 
surface  model  created  from  all  lidar  returns,  and  the  same  image  with  the  individual  lidar  points 
coded  black  for  ground  returns  and  green  for  non-ground  returns.  This  image  shows  an  area  of 
flight  line  overlap  on  the  left  side  of  the  image.  The  overlap  area  shows  a  higher  number  of 
green  (non-ground)  returns. 
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Figure  16:  Flight  Line  Overlap  Area  -  Pueblo  PBR 


Comparison  with  previous  data  sets.  As  with  the  previous  data  sets  examined,  the  density 
map  for  the  full  data  set  clearly  shows  the  effects  of  flight  line  overlap,  and  this  effect  is  largely 
absent  in  the  density  map  based  on  the  vendor’s  ground  points.  As  with  the  other  sites 
examined,  the  percentage  of  non-ground  returns  increased  roughly  with  overall  point  density. 

One  difference  from  the  three  data  sets  collected  by  Terra  Remote  Sensing  is  that  in  the  Pueblo 
data  examined  above,  the  maximum  all-points  density  is  lower:  22  pts/m2  compared  with  346 
pts/m2  in  the  Victorville  data.  This  may  be  because  the  lidar  data  at  Pueblo  was  collected  using 
a  fixed-wing  aircraft,  which  was  subject  to  less  pitching  due  to  wind  than  the  helicopter  used  at 
the  other  sites,  where  areas  of  aircraft  pitch  appear  to  account  for  the  very  high  data  densities 
occasionally  seen.  (The  lower  rate  of  aircraft  pitch  can  be  seen  in  the  all-points  density  map  as 
an  absence  of  “curtain  fold”  effects.) 

3.2.4  Former  Camp  Beale/Sky  Research 

The  ESTCP  data  for  the  Former  Camp  Beale  site  was  collected  by  Terra  Remote  Sensing; 
however,  Sky  Research  collected  lidar  data  at  Beale  AFB,  directly  to  the  west.  Sky  Research’s 
data  extended  a  small  distance  onto  the  Former  Camp  Beale  site,  providing  a  complementary 
view  of  the  same  area.  One  lidar  data  set  was  collected. 

Point  density  maps.  Figure  17  shows  point  density  for  the  full  point  data  set  and  for  ground 
points  alone. 
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Figure  17:  Point  Density  Maps  -  Former  Camp  Beale/Sky  Research 
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Test  areas.  Table  10  shows  the  results  for  10  test  areas,  each  50  m  x  50  m.  The  test  areas 
were  placed  to  avoid  tall  vegetation. 
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Table  10:  Former  Camp  Beale/Sky  Research  Test  Area  Results 


Area  ID 

Area  (m2) 

Number  of  Points 

Percentage 

Density  i 

pts/m2) 

Total 

Ground 

Non¬ 

ground 

Ground 

Non¬ 

ground 

All) 

Ground 

1 

2,500 

9976 

8444 

1532 

84.64 

15.36 

3.99 

3.38 

2 

2,500 

10654 

8316 

2338 

78.06 

21.94 

4.26 

3.33 

3 

2,500 

11435 

9794 

1641 

85.65 

14.35 

4.57 

3.92 

4 

2,500 

11739 

7121 

4618 

60.66 

39.34 

4.70 

2.85 

5 

2,500 

11755 

9903 

1852 

84.25 

15.75 

4.70 

3.96 

6 

2,500 

11814 

9329 

2485 

78.97 

21.03 

4.73 

3.73 

7 

2,500 

12011 

9489 

2522 

79.00 

21.00 

4.80 

3.80 

8 

2,500 

12052 

8507 

3545 

70.59 

29.41 

4.82 

3.40 

9 

2,500 

12140 

9854 

2286 

81.17 

18.83 

4.86 

3.94 

10 

2,500 

16667 

7793 

8874 

46.76 

53.24 

6.67 

3.12 

Figure  18:  Lidar  Points  on  Road  Surface  - 
Former  Camp  Beale/Sky  Research 


Comparison  with  previous  data  sets.  In  this  data  set,  the  overall  data  density  map  shows  the 
effect  of  flight  line  overlap  and  aircraft  motion.  Unlike  in  the  other  data  sets  examined,  these 
effects  persist  in  the  ground  point  density  model.  Based  on  conversations  with  Sky  Research 
staff,  this  may  be  a  result  of  more  successful  efforts  to  spatially  integrate  data  from  one  flight 
line  to  the  next,  resulting  in  less  spatial  displacement  (Sky  Research  2009). 


Table  10  shows  that  the  effects  observed  at  the  other  ESTCP  sites  are  less  evident.  While  the 
percentage  of  points  classified  as  non-ground  does  increase  with  overall  density,  this  effect  is 
much  less  pronounced  and  there  are  many  more  outliers.  The  percentage  of  returns  classified 
as  non-ground  does  vary  between  approximately  15  and  53  percent  in  areas  with  relatively 
consistent  vegetation  conditions.  Figure  18  shows  some  non-ground  points  on  a  paved  surface, 
indicating  that  although  the  effect  is  less  pronounced  than  at  the  other  ESTCP  sites,  there  are 
nevertheless  some  returns  from  the  paved  surface  being  classified  as  non-ground. 
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3.2.5  Puget  Sound  Lidar  Coalition,  Portland  Area 

Lidar  data  was  collected  for  this  area  by  TerraPoint  in  2004.  One  lidar  data  set  was  collected. 

Point  density  maps.  Figure  19  shows  point  density  for  the  full  point  data  set  and  for  ground 
points  alone. 


Figure  19:  Point  Density  Maps  -  Puget  Sound  Lidar  Coalition,  Portland  Area 


Test  areas.  Table  1 1  shows  the  results  for  10  test  areas,  each  just  under  30  m  x  30  m.  The 
test  areas  were  placed  on  roads  where  the  full  lidar  data  sets  showed  that  no  vehicles  were 
present. 
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Table  1 1 :  Puget  Sound  Lidar  Coalition,  Portland  Area  Test  Area  Results 


Area  ID 

Area 

(m2) 

Number  of  Points 

Percentage 

Density  | 

[pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non- 

Ground 

All 

Ground 

1 

883 

689 

666 

23 

96.66 

3.34 

0.78 

0.75 

2 

883 

691 

663 

28 

95.95 

4.05 

0.78 

0.75 

3 

883 

694 

666 

28 

95.97 

4.03 

0.79 

0.75 

4 

883 

695 

626 

69 

90.07 

9.93 

0.79 

0.71 

5 

883 

703 

671 

32 

95.45 

4.55 

0.80 

0.76 

6 

883 

706 

681 

25 

96.46 

3.54 

0.80 

0.77 

7 

883 

732 

639 

93 

87.30 

12.70 

0.83 

0.72 

8 

883 

1,036 

953 

83 

91.99 

8.01 

1.17 

1.08 

9 

883 

1,044 

971 

73 

93.01 

6.99 

1.18 

1.10 

10 

883 

1,102 

997 

105 

90.47 

9.53 

1.25 

1.13 

Comparison  with  previous  data  sets.  As  with  the  all  of  the  data  sets  examined,  the  effects  of 
flight  line  overlap  are  evident  in  the  overall  density  maps.  These  effects  are  still  somewhat 
evident  in  the  density  map  for  ground  points,  although  results  are  somewhat  complicated  by  the 
presence  of  trees  and  buildings  in  this  complex  suburban  area. 

The  classification  results  are  somewhat  different  from  the  ESTCP  sites.  First,  a  much  smaller 
percentage  of  returns  were  classified  as  non-ground,  and  while  this  percentage  does  increase 
with  overall  point  density,  there  are  many  more  outliers.  These  results  may  be  related  to  the 
lower  overall  point  density  of  this  data  set,  and  the  much  smaller  range  of  overall  point  density, 
from  0.78  to  1.25  pts/m2.  This  area  may  also  have  had  particularly  good  data  calibration 
between  flight  lines. 

3.2.6  Puget  Sound  Lidar  Coalition,  Snohomish  County 

Lidar  data  was  collected  for  this  area  by  TerraPoint  in  2003.  One  data  set  was  collected. 

Point  density  maps.  Figure  20  shows  point  density  for  the  full  point  data  set  and  for  ground 
points  alone. 
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Figure  20:  Point  Density  Maps  -  Puget  Sound  Lidar  Coalition,  Snohomish  County 


Test  areas.  Table  12  shows  the  results  for  10  test  areas,  each  just  over  20  m  x  20  m.  The  test 
areas  were  placed  on  roads  where  the  full  lidar  data  sets  showed  that  no  vehicles  were  present. 
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Table  12:  Puget  Sound  Lidar  Coalition,  Snohomish  County  Test  Area  Results 


Area  ID 

Area 

(m2) 

Number  of  Points 

Percentage 

Density  (pts/m2) 

Total 

Ground 

Non- 

Ground 

Ground 

Non- 

Ground 

All 

Ground 

1 

402 

338 

300 

38 

88.76 

11.24 

0.84 

0.75 

2 

402 

385 

225 

160 

58.44 

41.56 

0.96 

0.56 

3 

402 

404 

355 

49 

87.87 

12.13 

1.01 

0.88 

4 

402 

421 

323 

98 

76.72 

23.28 

1.05 

0.80 

5 

402 

489 

341 

148 

69.73 

30.27 

1.22 

0.85 

6 

402 

490 

315 

175 

64.29 

35.71 

1.22 

0.78 

7 

402 

511 

357 

154 

69.86 

30.14 

1.27 

0.89 

8 

402 

564 

377 

187 

66.84 

33.16 

1.40 

0.94 

9 

402 

605 

420 

185 

69.42 

30.58 

1.51 

1.05 

10 

402 

623 

391 

232 

62.76 

37.24 

1.55 

0.97 

Figure  21 :  Lidar  Points  on  Road  Surface  - 
Puget  Sound  Lidar  Coalition,  Snohomish  County 


Comparison  with  previous  data  sets.  As  with  all  of  the  lidar  data  sets  examined,  the  full  data 
sets  shows  the  effects  of  flight  line  overlap.  This  effect  is  largely  absent  from  the  density  map 
for  ground  points.  This  may  be  an  effect  of  heavy  vegetation  and  buildings  in  this  complex 
suburban  area.  As  with  the  Portland  area  data,  the  range  of  overall  lidar  density  is  lower  than 
for  the  ESTCP  sites,  from  0.84  to  1 .55  pts/m2. 


While  the  number  of  non-ground  returns  is  lower  than  for  the  ESTCP  sites,  it  remains  a 
significant  portion  (often  over  30  percent)  of  the  overall  lidar  points,  given  that  the  test  areas  are 
on  roads  with  no  vegetation.  However,  the  correlation  between  overall  lidar  density  and  the 
percentage  of  points  classified  as  non-ground  is  less  clear  than  for  the  ESTCP  sites.  At  both 
this  site  and  the  Portland  site,  the  weaker  correlation  may  be  the  result  of  the  lower  overall 
density. 
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3.3  Analysis 


Based  on  discussions  with  processing  staff  at  the  three  vendors  contacted,  it  appears  that  the 
large  number  of  returns  classified  as  non-ground  in  high  density  and  flight  line  overlap  areas  is 
likely  the  result  of  the  operation  of  TerraSolid’s  algorithms,  interacting  with  small  vertical  errors 
between  lidar  points. 


In  part,  this  effect  results  from  the  fact  that  when  point  density  is  higher,  the  lines  of  laser  points 
will  be  closer  together,  as  shown  in  Figure  22. 


Figure  22:  Lidar  Point  Spacing  Differences 


1.5m. 


Higher  density  lidar,  average  point  spacing  of  0.2m  x 
0.4m. 


When  laser  points  are  closer  together,  the  angle  formed  by  any  new  point  with  the  existing 
model  will  be  larger.  When  laser  returns  are  sufficiently  close,  this  increase  in  angle  can  be 
large  enough  so  that  very  small  elevation  variations,  within  the  noise  level  of  the  equipment,  can 
cause  points  to  be  classified  as  non-ground. 


Figure  23  shows  the  difference  in  iteration  angle  that  results  from  decreasing  the  distance 
between  the  lines  of  points  in  this  manner.  Iteration  angle  was  calculated  for  lines  of  laser 
points  100  cm  apart  vs.  20  cm  apart,  with  elevations  of  15,  10,  and  5  cm  above  the  initial 
surface.  These  elevations  were  chosen  because  the  stated  lidar  vertical  accuracy  of  lidar  points 
is  typically  15  cm,  the  typical  flight  line-to-flight  line  error  is  around  10  cm,  and  the  elevation 
difference  between  sequential  lines  of  lidar  points  is  often  around  5  cm. 


All  of  the  vendors  contacted  believe  that  this  effect  accounts  for  the  large  number  of  points 
classified  as  non-ground.  This  conclusion  is  consistent  with  the  results  described  earlier, 
showing  that  as  overall  density  increased,  the  classification  algorithms  classified  an  increasing 
percentage  of  points  as  non-ground.  This  also  may  account  for  the  weaker  effects  seen  in  the 
Puget  Sound  Lidar  Coalition  data,  which  was  acquired  at  lower  overall  densities. 
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Figure  23:  Example  Iteration  Angles 
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The  over-classification  of  non-ground  returns  does  not,  in  itself,  address  the  issue  of  the  source 
of  the  vertical  offsets  which  with  the  classification  software  is  interacting.  One  vendor  contacted 
pointed  out  that  misclassification  would  be  a  predictable  effect  of  errors  in  GPS  or  flight  line-to- 
flight  line  data  calibration,  either  of  which  would  result  in  larger  than  necessary  vertical 
discrepancies  between  points.  This  vendor  noted  that  if  the  flight  lines  are  vertically  offset  (due 
to  calibration  or  GPS  errors),  then  points  associated  with  the  higher  flight  line  would  be 
predictably  classified  as  non-ground.  This  effect  would  be  present  even  if  the  classification 
routine  were  well  constructed.  (Sky  Research,  2009) 

Reducing  the  misclassification  effect  should  therefore  include  checks  for  successful  calibration 
and  GPS  quality.  Striations  or  “corduroy”  effects  in  the  lidar  surface  models  (as  in  Figure  24) 
may  be  another  indicator  of  poor  calibration,  poor  GPS  data,  or  a  noisy  platform  and  should  be 
cause  for  further  checks. 

However,  better  calibration  will  not  eliminate  the  inherent  error  between  individual  lidar  points  or 
lines  of  lidar  points,  which  may  lead  to  some  misclassification  depending  on  the  parameters  set 
up  in  the  classification  routine  and  the  distance  between  the  laser  returns.  Consequently, 
careful  examination  of  point  classification  methods  will  remain  desirable  whenever  lidar  is  used 
to  detect  and  discriminate  small  ground  features. 
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4  Results  of  Increasing  Ground  Point  Density  on  Feature  Detection 

This  section  examines  the  results  of  a  preliminary  reclassification  of  the  lidar  points  to  add  more 
of  the  “non-ground”  points  back  into  the  lidar  surface  model.  The  objective  of  this  test  was  not 
to  develop  a  recommended  reclassification  method.  Rather,  this  test  was  intended  to  determine 
whether  adding  the  maximum  possible  number  of  points  back  into  the  surface  model  would 
improve  feature  detection  enough  to  warrant  further  investigation. 

4.1  Potential  Reclassification  Methods 

TerraScan  allows  two  straightforward  approaches  to  changing  point  classification.  First, 
iteration  distance  could  be  increased  and/or  iteration  angle  could  be  lowered,  and  second,  all 
points  within  a  given  distance  of  the  initial  surface  could  be  classified  as  ground. 

Either  of  these  approaches  would  classify  more  points  as  ground  returns,  and  could  increase 
the  resolution  of  the  resulting  surfaces.  Both  would  add  noise  due  to  small  measurement  errors 
and  low  vegetation.  Noise  effects  have  been  documented  in  earlier  work,  and  have  been  seen 
in  lidar  models  of  smooth  ground  surfaces  at  the  ESTCP  WAA  demonstration  sites,  as  shown  in 
Figure  24. 


Figure  24:  Noise  Effects  in  Lidar  Surface  Models 


Former  Camp  Beale  site.  Surface  noise  appears  as  “corduroy”  effect.  Addition  of  the  lidar  points  to  the  image  (right) 
shows  that  the  “corduroy”  lines  are  associated  with  the  small  elevation  differences  (approximately  5  cm)  between  the 
lines  of  lidar  points. 


4.2  Reclassification  Test 

Methodology.  For  this  initial  test,  59  test  areas  were  established  where  the  original  lidar 
surface  models  showed  ground  features  such  as  individual  potential  craters.  In  all  but  three  of 
these  plots,  the  identified  features  were  either  completely  or  partially  under  tree  cover.  Features 
under  tree  cover  were  chosen  since  this  is  where  reclassification  would  have  the  most  utility. 
(Features  in  the  open  can  always  be  examined  using  the  full  lidar  data  set.) 

The  lidar  points  at  the  Former  Camp  Beale  site  were  reclassified  such  that  all  points  within  40 
cm  of  the  ground  surface  using  the  vendor-classified  points  were  reclassified  as  ground  points. 
This  distance  was  chosen  based  on  specific  conditions  of  this  site  to  include  as  many  points  as 
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possible  without  adding  higher  vegetation.  These  operations  were  performed  using  ArcGIS, 
and  mimic  the  similar  process  in  TerraScan. 

Lidar  surface  models  were  created  using  the  reclassified  points.  The  surface  models  using  the 
reclassified  points  were  compared  to  those  made  using  the  vendor’s  original  classification.  The 
aim  of  the  examination  was  to  determine  1 )  whether  the  existing  features  showed  more  clearly 
in  the  reclassified  surface,  and  2)  whether  additional  features  were  visible. 

Results.  New  surface  models  using  the  reclassified  points  showed  the  previously-identified 
features  in  somewhat  greater  detail,  and  revealed  potential  additional  features  at  21  of  the  59 
plots  (36  percent).  It  should  be  emphasized  that  these  potential  features  were  all  small,  and 
none  were  shown  with  great  clarity.  Some  may  be  debris  lying  on  the  ground.  These  additional 
features  did,  however,  indicate  areas  where  additional  investigation  would  be  warranted.  A 
representative  sample  of  the  initial  results  is  shown  in  Figure  25.  The  full  data  set,  including 
orthophotos  for  each  of  the  test  areas,  is  shown  in  Appendix  A. 


Figure  25:  Surface  Models  Using  Reclassified  Points  -  Initial  Comparisons 
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Figure  25:  Surface  Models  Using  Reclassified  Points  -  Initial  Comparisons  (continued) 
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Plot  6:  Vendor-classified  points. 


Plot  6:  Re-classified  points. 
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As  a  second  test,  the  surface  models  for  nine  test  areas  were  re-created  using  a  0.3  m  (1-foot) 
cell  size  rather  than  the  original  1.0  m  cell  size  used  for  all  of  the  ESTCP  demonstration  sites. 
Surface  models  can  be  created  with  cells  of  any  size,  and  smaller  cells  will  yield  more  sharply- 
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defined  features.  This  ability  is  limited  by  the  density  of  the  underlying  data,  and  in  some  cases 
the  additional  detail  can  be  misleading. 

New  surface  models  were  created  using  both  the  vendor’s  classification  and  the  reclassified 
data.  In  this  test,  the  0.3  m  cells  showed  the  features  with  more  clarity,  as  expected.  The 
surfaces  created  with  the  reclassified  points  showed  additional  small  features  in  the  same 
manner  as  those  at  the  1 .0  m  cell  size.  The  pattern  of  additional  small  features  detected  was 
somewhat  different,  with  more  features  detected  using  the  smaller  cell  size  (Figure  26).  Some 
of  the  additional  features  appeared  to  be  downed  tree  trunks,  although  most  could  not  be  clearly 
identified. 


Figure  26:  Surface  Models  Using  Reclassified  Points  - 
Comparisons  Using  Smaller  Cell  Sizes 
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Plot  2:  Vendor-classified  points,  0.3m  grid  size.  Plot  2:  Reclassified  points,  0.3m  grid  size, 


Figure  26:  Surface  Models  Using  Reclassified  Points 
Comparisons  Using  Smaller  Cell  Sizes  (continued) 


Plot  3:  Vendor-classified  points,  0.3m  grid  size.  Plot  3:  Reclassified  points,  0.3m  grid  size 
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Figure  26:  Surface  Models  Using  Reclassified  Points  - 
Comparisons  Using  Smaller  Cell  Sizes  (continued) 
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Figure  26:  Surface  Models  Using  Reclassified  Points  - 
Comparisons  Using  Smaller  Cell  Sizes  (continued) 


Plot  7:  Vendor-classified  points,  0.3m  grid  size. 


Plot  7:  Reclassified  points,  0.3m  grid  size. 
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A  third  comparison  was  made  between  the  vendor-classified  and  reclassified  data  by  changing 
the  process  for  creating  the  ground  surface  models.  At  all  of  the  ESTCP  demonstration  sites, 
surface  models  were  created  by  first  creating  a  TIN  using  every  laser  return  as  the  vertex  of  the 
triangles.  Surface  models  were  then  created  from  the  TINs.  This  method  was  chosen  since  it 
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uses  every  available  laser  return  to  create  the  initial  model  from  which  the  surfaces  are  derived. 
The  alternative  to  this  method  is  to  use  one  of  several  methods  to  mathematically  interpolate 
surfaces  directly  from  the  points. 

Several  interpolation  approaches  were  compared  to  the  TIN  method  during  the  Kirtland  AFB 
PBR  demonstration  site  investigation,  and  no  significant  difference  was  found  (URS  2007b). 

For  the  Former  Camp  Beale  data,  surfaces  were  created  for  three  test  areas  using  ArcGIS 
Inverse  Distance  Weighted  interpolation  method,  using  the  default  settings.  This  comparison 
was  repeated  using  a  1 .0  m  and  a  0.3  m  cell  size. 

Unlike  the  Kirtland  data,  the  interpolated  surfaces  showed  more  features  than  the  TIN-derived 
surfaces  at  two  of  the  three  test  areas.  The  reason  for  this  difference  between  the  Former 
Camp  Beale  data  and  the  Kirtland  data  is  not  clear,  and  further  investigation  would  be 
warranted. 

As  with  the  TIN-derived  surfaces,  using  the  reclassified  points  showed  the  existing  features 
more  clearly,  and  showed  additional  small  features  (Figure  27).  Changing  the  cell  size  to  0.3  m 
resulted  in  clearer  definition  of  the  features  that  were  shown  (Figure  28) 


Figure  27:  Surface  Models  Using  Reclassified  Points  - 
Comparisons  Using  Interpolated  Surfaces,  1.0  m  cell  size 


Plot  1 :  Vendor  classified  points,  TIN-derived  surface.  Plot  1 :  Reclassified  points,  TIN-derived  surface 

Repeated  from  Figure  26.  Repeated  from  Figure  26. 
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Plot  1:  Vendor  classified  points,  interpolated  surface. 

The  interpolated  surface  shows  some  features  not  visible 
in  the  TIN-derived  surface  above. 


Plot  1:  Reclassified  points,  interpolated  surface.  The 
interpolated  surface  gives  a  different  interpretation  of  the 
three  small  features  shown  in  the  TIN-derived  surface 
above. 


Figure  27:  Surface  Models  Using  Reclassified  Points  - 
Comparisons  Using  Interpolated  Surfaces,  1.0  m  cell  size  (continued) 


Plot  2:  Vendor  classified  points,  TIN-derived  surface. 
Repeated  from  Figure  26. 


Plot  2:  Reclassified  points,  TIN-derived  surface. 
Repeated  from  Figure  26. 
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Plot  2:  Vendor  classified  points,  interpolated  surface.  Plot  2:  Reclassified  points,  interpolated  surface 

The  TIN  and  interpolated  surfaces  are  similar. 


Figure  27:  Surface  Models  Using  Reclassified  Points  - 
Comparisons  Using  Interpolated  Surfaces,  1.0  m  cell  size  (continued) 


Plot  3:  Vendor  classified  points,  TIN-derived  surface.  Plot  3:  Reclassified  points,  TIN-derived  surface 

Repeated  from  Figure  26.  Repeated  from  Figure  26. 
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Plot  3:  Vendor  classified  points,  interpolated  surface. 

The  interpolated  surface  shows  features  not  visible  in  the 
TIN-derived  surface  above. 


Plot  3  -  Reclassified  points,  interpolated  surface.  As  with 
the  TIN-derived  surface  above,  the  interpolated  surface 
shows  what  appears  to  be  down  trees. 
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Figure  28:  Surface  Models  Using  Reclassified  Points  - 
Comparisons  Using  Interpolated  Surfaces,  0.3  m  cell  size 


Plot  1:  Vendor  classified  points,  TIN-derived  surface,  Plot  1:  Reclassified  points,  TIN-derived  surface,  0.3m  cell 

0.3m  cell  size.  Repeated  from  Figure  27.  size.  Repeated  from  Figure  27. 
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Plot  2:  Vendor  classified  points,  TIN-derived  surface,  Plot  2:  Reclassified  points,  TIN-derived  surface,  0.3m  cell 

0.3m  cell  size.  Repeated  from  Figure  27.  size.  Repeated  from  Figure  27. 


Plot  2:  Vendor  classified  points,  interpolated  surface,  Plot  2:  Reclassified  points,  interpolated  surface,  0.3m  cell 

0.3m  cell  size.  As  with  the  1 .0m  cells,  the  TIN  and  size, 

interpolated  surfaces  are  similar. 
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Plot  3:  Vendor  classified  points,  TIN-derived  surface,  Plot  2:  Reclassified  points,  TIN-derived  surface,  0.3m  cell 

0.3m  cell  size.  Repeated  from  Figure  27.  size.  Repeated  from  Figure  27. 


Plot  3:  Vendor  classified  points,  interpolated  surface,  Plot  3:  Reclassified  points,  interpolated  surface,  0.3m  cell 

0.3m  cell  size.  As  with  the  1 .0m  cells,  the  interpolated  size, 
surface  shows  features  not  visible  in  the  TIN-derived 
surface. 
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A  final  comparison  was  made  between  the  all-points  and  ground-points  surface  models  in  the 
area  of  the  1 .5  and  0.3  m  test  craters  that  were  established  as  part  of  the  lidar  investigation  at 
the  Former  Camp  Beale  site.  The  test  craters  were  established  in  an  area  with  little  to  no 
vegetation,  so  reclassification  would  be  equivalent  to  using  the  full  data  set.  As  with  the  other 
comparisons,  the  features  were  somewhat  more  clearly  defined  using  the  reclassified  points 
and  the  surface  included  more  noise  (the  corduroy  lines).  However,  no  additional  features  were 
seen  (Figure  29). 

Figure  29:  Surface  Model  Compound:  Test  Craters 


Former  Camp  Beale  test  craters,  surface  model  based  Former  Camp  Beale  full  lidar  data  set.  Features  are  more 
on  vendor-classified  ground  points.  clearly  defined,  but  no  additional  features  are  seen. 

Noise  effects  are  more  visible. 
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5  Conclusions 

Lidar  has  been  shown  to  be  an  effective  tool  for  UXO/MEC  WAA.  The  focus  of  this  investigation 
is  to  determine  whether  adjustments  to  the  point  classification  can  increase  the  detection 
capability  and  thereby  increase  the  effectiveness  of  this  technology.  Preliminary  results  indicate 
that  some  increase  in  detection  capability  can  be  achieved. 

At  the  Former  Camp  Beale  demonstration  site,  large  numbers  of  laser  returns  from  the  ground 
surface  were  classified  by  the  vendor  as  non-ground.  This  effect  correlated  strongly  with  the 
density  of  the  lidar  points.  In  local  areas  of  very  high  lidar  point  density,  over  90  percent  of  the 
points  were  classified  as  non-ground,  even  on  paved  road  surfaces.  The  effect  appears  to  be 
the  result  of  commonly-used  point  classification  algorithms  interacting  with  local  areas  of  high 
lidar  data  density. 

This  interaction  of  the  point  classification  methods  with  increased  lidar  point  density  has  the 
effect  of  reducing  the  effectiveness  of  newer,  faster  sensors.  Newer  equipment  produces  a 
higher  number  of  laser  pulses  per  second,  but  at  least  on  some  sites,  much  of  this  additional 
data  could  be  classified  as  non-ground. 

The  practical  impact  of  this  over-classification  of  non-ground  points  will  depend  on  the  site.  At 
desert  sites  with  little  vegetation,  the  effect  will  be  smaller  since  the  ground  surface  can  be 
modeled  using  the  full  lidar  data  set.  At  sites  with  more  vegetation,  the  ground  surface  must  be 
modeled  using  the  ground  returns  in  order  to  eliminate  vegetation  (or  buildings),  and  the  effect 
will  be  more  pronounced. 

The  effect  can  be  eliminated  by  changing  the  settings  in  the  standard  classification  software. 
Doing  so  would  add  additional  points  to  the  ground  surface  model,  along  with  additional  noise. 
An  additional  approach,  not  tested  in  this  paper,  would  be  to  reduce  the  vertical  discrepancies  in 
the  lidar  data  closer  to  the  inherent  noise  level.  It  is  logical  to  suspect  that  tighter  calibration, 
especially  between  flight  lines,  could  reduce  misclassification  and  result  in  a  larger  number  of 
points  classified  as  ground,  even  without  changing  the  point  classification  routine. 

A  preliminary  test  of  reclassifying  the  lowest  points  as  ground  returns  showed  that 
reclassification  may  reveal  some  additional  features.  These  additional  features  are  all  small, 
since  the  vendor-classified  ground  points  are  sufficiently  dense  to  reveal  larger  features.  Most 
of  these  features  could  not  be  identified,  and  some  appeared  to  be  debris  such  as  down  tree 
trunks. 

These  results  indicate  that  adjustment  of  the  vendor’s  classification  methods  to  increase  the 
density  of  the  surface  model  would  be  worthwhile.  This  is  especially  the  case  since  such 
adjustment  would  not  increase  processing  time  or  cost.  The  reclassification  test  also  showed 
that  methods  used  to  create  the  surface  models  may  be  more  relevant  to  feature  detection  than 
shown  in  previous  investigations. 

Because  lidar  data  sets  are  very  large,  many  vendors  only  deliver  derived  products  such  as 
ground  surface  models  (DEMs)  and  all-points  models  (DTMs).  The  results  of  this  investigation 
suggest  that  the  Government  should  receive  the  entire  lidar  point  data  set  for  all  lidar 
investigations.  Using  this  data,  Government  land  managers  can  evaluate  point  density  patterns, 
apparent  calibration  success  and  the  approaches  to  point  classification  to  surface  model 
creation  used  by  the  vendor,  and  make  appropriate  adjustments.  T 
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1  Introduction  to  Lidar  and  Orthophotography 

1.1  Objectives 

This  guidance  document  is  intended  to  assist  Government  site  managers  who  are 
considering  the  use  of  light  detection  and  ranging  (lidar)  and  orthophotography  at 
munitions  sites.  The  document  summarizes  lessons  learned  and  describes  suggested 
contract  specifications  and  analysis  methods  for  these  technologies. 

Lidar  and  orthophotography  are  airborne  technologies  that  can  be  used  to  give  a  rapid 
assessment  of  site  conditions.  In  the  context  of  munitions  management,  lidar  and 
orthophotos  have  been  used  to  locate  surface  features  that  may  indicate  historic 
munitions  use  including  bombing  targets,  crater  fields,  open  burning/open  detonation 
(OB/OD)  areas,  and  smaller  individual  items  such  as  craters,  bunkers,  and  firing  points. 

Lidar  and  orthophotos  are  most  appropriately  used  at  the  beginning  of  the  site 
assessment  process.  These  technologies  are  capable  of  assessing  large  areas  for 
which  historical  data  may  be  incomplete,  providing  an  additional  layer  of  data  that  can 
corroborate  or  correct  historical  records.  The  data  may  be  used  to  improve  the  accuracy 
of  the  conceptual  site  model  (CSM)  and  to  prioritize  and  focus  the  use  of  more 
expensive  technologies  that  directly  detect  munitions  components. 

1 .2  Overview  of  Lidar  and  Orthophoto  Technologies 
1.2.1  Lidar 

Airborne  lidar  is  a  well-established  technology  for  modeling  ground  surfaces.  The 
technology  was  first  developed  in  the  late  1960s  and  early  1970s,  and  has  been  used  for 
modeling  ground  surfaces  since  the  mid  1980s.  Lidar  has  been  in  wide  commercial  use 
since  the  early  1990s,  and  the  accuracies  and  limitations  of  lidar  for  surface  modeling 
are  well  documented. 

Lidar  is  based  on  measuring  the  time  of  return  for  a  laser  pulse  from  the  laser  to  the 
sensor.  The  lidar  system  is  mounted  in  a  helicopter  or  small  aircraft,  and  Global 
Positioning  System  (GPS)  and  Inertial  Measurement  Unit  (IMU)  technology  are  used  to 
locate  the  sensor  in  the  air.  Measuring  the  time  of  return  allows  for  the  accurate 
calculation  of  the  point  of  reflection  of  the  laser  signal  from  the  ground,  buildings,  or 
vegetation  (Figure  1 ). 

When  the  laser  pulse  approaches  the  ground  it  is  wide  enough  that  part  of  the  pulse  may 
reflect  from  an  object,  while  the  remainder  of  the  pulse  continues  to  the  ground  or 
another  object.  Sensors  can  detect  these  multiple  reflections  from  a  single  laser  pulse, 
which  increases  the  probability  of  sampling  the  ground  surface  through  gaps  in 
vegetation.  Automated  algorithms  followed  by  operator  inspection  are  used  to 
categorize  each  data  point  as  returning  from  the  ground  surface,  vegetation,  structures 
or  other  objects. 

Lidar  points  have  a  vertical  positional  accuracy  of  approximately  15  cm  and  a  horizontal 
accuracy  of  approximately  60  cm,  both  compared  to  surveyed  control  points.  Accuracy 
will  be  greater  at  lower  altitudes  and  on  relatively  flatter  and  harder  surfaces. 
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Figure  1 :  Lidar  System  Operations 


Once  elevation  data  are  collected  in  the  form  of  lidar  points,  surface  models  are  created 
and  analyzed,  and  maps  and  analysis  products  generated.  Although  specialized 
software  now  exists,  most  analysis  of  lidar  data  can  be  conducted  using  standard 
Geographic  Information  Systems  (GIS)  software  and  methods,  and  some  of  the  process 
can  be  automated. 

Lidar  has  been  become  a  standard  tool  for  many  applications  that  require  detailed 
ground  surface  models,  including  seismic  and  geohazard  studies;  corridor  and  siting 
studies  for  pipelines,  power  lines,  roads,  and  railroads;  floodplain  and  other  hydrology 
studies;  and  land  management  programs.  The  popularity  of  lidar  is  due  to  its  ability  to 
provide  more  accurate  topographic  data  than  other  typically-available  sources  (such  as 
the  US  Geological  Survey  [USGS]  maps  and  digital  data)  at  a  reasonable  cost  compared 
to  alternatives  such  as  ground  survey  or  photogrammetry.1  Each  application  of  lidar  will 
have  appropriate  specifications  for  data  collection,  processing  and  analysis. 

Investigation  of  munitions  sites  using  lidar  and  orthophotography  will  similarly  call  for 
appropriate  specifications  that  may  differ  from  other  applications. 

1.2.2  Orthophotography 

An  orthophotograph  is  a  digital  aerial  photograph  that  has  been  geometrically  corrected 
for  topographic  relief,  lens  distortion,  and  camera  tilt.  Individual  digital  images  are  first 
combined  into  a  mosaic,  and  then  the  composite  image  is  spatially  corrected  using 
sophisticated  mathematical  algorithms  including  the  use  of  terrain  data,  such  as  data 
from  lidar.  This  process,  called  orthorectification,  allows  for  the  accurate  spatial  location 
of  each  photo  pixel,  and  this  allows  the  images  to  be  used  in  a  GIS  or  computer-aided 
design  and  drafting  (CADD)  system  with  other  spatial  data  such  as  contour  lines,  survey 
data,  or  the  results  of  other  investigations  such  as  magnetometry.  While  orthophotos 
are  adjusted  so  that  the  pixels  are  spatially  accurate,  this  only  refers  to  pixels 
representing  the  ground  surface.  Pixels  representing  the  tops  of  buildings  or  tall  trees 
can  be  slightly  displaced. 


1  For  instance,  a  recent  paper  examined  USGS  10-m  digital  elevation  data  for  the  area  near  Seattle, 
Washington,  and  concluded  that  errors  in  the  digital  data  would  have  a  significant  effect  on  standard 
equations  for  earthquake-triggered  landslide  assessment  (Haneberg  2006). 
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Digital  orthophotography  has  been  commercially  available  since  the  early  1980s,  with 
steady  improvement  in  the  resolution  (i.e.,  pixel  size)  and  precision  (i.e.,  pixel 
placement)  of  the  images  as  the  technology  of  digital  cameras,  GPS,  and  IMU  systems 
has  advanced.  Digital  cameras  have  largely  replaced  film  cameras  in  the  production  of 
orthophotos  (Dold  2008). 

Airborne  digital  cameras  have  been  integrated  with  lidar  sensors  successfully,  and 
because  the  cameras  and  lidar  sensors  use  the  same  GPS  and  Inertial  Navigation 
System  (INS),  the  two  data  sets  can  be  accurately  integrated.  Vendors  generally 
guarantee  a  horizontal  accuracy  of  three  pixel  widths  compared  to  ground  control  for 
orthophotography,  and  spatial  integration  within  two  pixel  widths  for  orthophotos  and 
lidar. 

Final  orthophoto  pixel  size  depends  on  flight  altitude  and  camera  specifications.  For 
munitions  sites,  orthophoto  pixel  size  is  typically  from  0.3  m  (about  1  foot)  down  to  0.1  m 
(about  4  inches)  (Nelson  et  al.  2008).  Smaller  pixel  sizes  than  this  are  generally 
impractical  due  to  the  low  flight  elevations  and  slow  flight  speeds  required  to  collect 
properly  overlapping  images,  and  the  large  numbers  of  images  that  would  need  to  be 
combined  into  the  mosaic.  Many  areas  also  have  existing  orthophotography  available  at 
lower  resolution,  with  pixels  in  the  0.5  m  -  1 .0  m  range.2 

1.3  System  Components 

1.3.1  Lidar 

Lidar  systems  are  manufactured  by  several  vendors  worldwide,  the  largest  of  which  are 
Optech,  based  in  Canada,  and  Leica,  based  in  Switzerland  and  the  US.  Other  vendors 
include  IGI  Ltd.  in  Germany  and  Laseroptronix  in  Sweden.  In  addition,  some  lidar 
vendors  assemble  their  own  systems  using  standard  components.  Since  the  1990s, 
lidar  system  development  has  primarily  focused  on  increasing  laser  pulse  rate,  adding 
the  ability  to  receive  multiple  returns  from  each  laser  pulse,  and  recording  the  intensity  of 
each  return.  Each  lidar  system  has  slightly  different  performance  specifications; 
however,  both  commercial  off-the-shelf  systems  and  vendor-built  systems  appear  to 
function  reliably  and  to  meet  their  published  specifications  if  used  and  functioning 
correctly.  A  typical  lidar  system  is  shown  in  Figure  2. 

Lidar  system  components  are  designed  to  determine  the  position  of  the  aircraft  and  the 
distance  and  angle  to  the  point  of  reflection  of  the  laser  pulse.  The  position  of  the 
aircraft  is  determined  using  GPS  and  INS.  GPS  uses  the  position  of  at  least  four  orbiting 
navigation  satellites  to  sample  the  aircraft  location.  Measurements  are  taken  between 
one  and  ten  times  per  second,  depending  on  the  manufacturer  and  model  in  use 
(Terrapoint,  2009).  Vendors  report  that  error  in  the  location  of  the  aircraft  is  primarily 
GPS  error,  and  that  the  most  serious  source  of  GPS  error  is  momentary  loss  of  satellite 
data  sufficient  to  establish  locations  accurately  (Terra  Remote  Sensing,  2008). 


2  States,  cities  and  counties  acquire  orthophotos  for  a  variety  of  purposes,  not  generally  including  detection 
of  smaller  objects.  Consequently,  these  entities  generally  do  not  incur  the  considerably  larger  costs  of 
higher-resolution  orthophotos.  These  coarser  images  can  nevertheless  be  useful  on  munitions  sites  with 
tree  cover,  where  individual  features  are  not  visible  and  the  primary  use  of  the  images  will  be  overall  mission 
planning. 
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Figure  2:  Helicopter-Mounted  Lidar  and  Orthophoto  System 


GPS  antenna  and  lidar  and  orthophoto  sensor  pod. 
Images  courtesy  of  Terra  Remote  Sensing,  Inc. 


Operator  control  panel, 
system  control  and  backup 
components. 


The  INS  calculates  the  position  and  orientation  of  the  aircraft  between  GPS  locations. 
The  primary  input  to  the  INS  is  from  the  IMU,  which  detects  and  records  the  rate  of 
acceleration  along  with  changes  in  rotational  attributes  such  as  pitch,  roll,  and  yaw.  The 
IMU  generally  contains  three  accelerometers  and  three  gyroscopes,  placed  in 
orthogonal  positions  so  that  data  is  collected  in  all  three  planes.  Current  IMUs  record 
these  changes  at  100  -  2,000  hertz  (Hz)  (samples  per  second).  Small  errors  in  the  IMU 
positions  accumulate  due  to  IMU  drift  as  positions  are  continually  re-calculated  until  the 
aircraft’s  position  is  updated  from  the  GPS. 


The  lidar  sensor  system  consists  of  a  laser,  a  receiver,  and  a  mirror  that  directs  each 
pulse  toward  the  ground  surface.  Lidar  systems  typically  use  lasers  with  wavelengths  of 
1 ,000  to  1 ,500  nanometers.  The  mirror  system  may  be  rotating  or  oscillating,  depending 
on  the  manufacturer. 


The  receiver  is  a  passive  device  that  is  tuned  to  the  frequency  of  the  laser,  which 
records  a  signal  when  the  amplitude  of  light  in  that  frequency  exceeds  a  threshold  value. 
The  threshold  value  that  will  trigger  a  return  is  typically  set  by  the  manufacturer  and  is 
not  adjustable.  The  position  of  the  laser  reflection  is  calculated  using  the  time  of  return 
combined  with  the  known  angle  of  the  pulse. 

The  sensor  also  records  the  energy  level  of  the  return,  which  is  referred  to  as  its 
intensity  value.  This  value  will  be  higher  from  more  reflective  surfaces.  Intensity  values 
can  be  used  to  create  quick  and  spatially  accurate  black-and-white  images  of  the 
surface  (Figure  3),  and  can  be  used  to  detect  some  site  conditions  that  are  not  based  on 
elevation,  such  as  the  boundary  between  paved  and  unpaved  surfaces. 
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Figure  3:  Lidar  Intensity  Image 


Lidar  intensity  image,  with  grey-scale  values  based  on 
the  energy  of  the  lidar  return.  More  reflective  surfaces 
are  lighter. 


The  sensor  can  record  multiple  returns  from  each  laser  pulse.  In  vegetated  conditions, 
part  of  the  laser  return  may  be  reflected  from  branches  and  other  features,  leaving  the 
remainder  of  the  signal  to  be  reflected  from  the  ground  surface  (Figure  4).  The  limitation 
of  the  multi-return  capability  is  that  there  must  be  sufficient  time  between  returns  for  the 
sensor  to  reset.  This  time  is  generally  between  three  and  20  nanoseconds  depending 
on  the  sensor  used,  a  distance  of  between  0.5  m  and  3  m.  Thus,  multiple  returns  cannot 
be  recorded  in  low  vegetation. 

Figure  4:  Lidar  Multi-Return  Capability 
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A  relatively  recent  development  in  laser  component  technology  is  the  analog  or  “full 
wave  form”  laser  receiver.  In  contrast  to  the  approach  described  above,  the  analog 
receiver  records  a  continuous  level  of  energy  values  once  the  amplitude  passes  the 
threshold  value.  The  shape  of  this  energy  return  curve  can  be  analyzed  and  multiple 
return  values  derived  in  a  more  interactive  manner.  Analog  receivers  create  larger  data 
sets  than  traditional  sensors,  and  more  advanced  software  is  needed  to  process  the 
output.  The  number  of  vendors  using  analog  receivers  was  small  but  increasing  at  the 
time  of  this  report. 

Lidar  systems  in  current  use  also  include  bathymetric  lidar  systems  that  can  map 
shallow  underwater  surfaces.  Bathymetric  lidar  systems  use  a  blue-green  laser  signal 
that  will  penetrate  water,  often  in  conjunction  with  a  red  wavelength  laser  that  reflects 
from  the  water  surface.  The  time  difference  between  the  two  signals  is  used  to 
determine  water  depth  (Optech  2008).  Bathymetric  lidar  systems  are  capable  of 
measuring  water  depths  to  a  maximum  of  approximately  50  m;  however,  success  of  the 
system  is  limited  by  many  factors,  including  water  turbidity  and  the  reflectivity  of  the 
bottom  surface.  Bathymetric  lidar  systems  are  less  commonly  used  than  terrestrial 
systems  and  costs  tend  to  be  considerably  higher.  However,  where  applicable, 
bathymetric  lidar  can  provide  data  on  the  nearshore  environment  where  the  water  is  too 
shallow  to  map  using  sonar  or  other  on-water  geophysical  survey  methods. 

The  final  components  of  the  sensor  system  are  the  power  source,  the  hardware-  and 
software-based  control  system,  and  the  data  storage  equipment  (consisting  of  multiple 
high-speed  hard  drives). 

The  lidar  system  has  some  parameters  that  can  be  varied  within  the  equipment 
specifications  of  each  manufacturer.  These  include: 

■  Power  level.  The  power  of  each  laser  pulse  is  adjustable  up  to  the  maximum 
power  of  the  system.  Maximum  power  levels  are  set  by  the  manufacturer  to  be 
safe  for  humans  and  wildlife. 

■  Pulse  rate.  The  maximum  pulse  rate,  or  the  number  of  laser  signals  per  second, 
has  increased  steadily  as  the  technology  has  evolved,  from  4  to  10  kilohertz 
(kHz)  in  the  mid-1990s  to  as  much  as  250  kHz  at  the  time  of  this  report.  Pulse 
rates  are  typically  adjustable. 

■  Mirror  speed.  The  oscillation  or  rotation  speed  of  the  mirror  can  be  controlled 
independently  of  the  laser  pulse  rate. 

■  Scan  angle.  The  maximum  angle  of  the  mirror  off  nadir  is  adjustable.  In 
combination  with  the  aircraft  altitude,  the  scan  angle  controls  the  width  of  the 
data  collection  swath. 

Vendors  report  that  the  primary  parameters  that  are  varied  are  pulse  rate  and  scan 
angle. 

In  theory,  higher  pulse  rate  systems  should  provide  more  laser  reflections  and  therefore 
more  detailed  surface  models.  However,  the  relationship  of  laser  pulse  rate  to  system 
performance  is  not  always  straightforward.  This  is  because  the  power  system  of  the 
laser  is  finite,  and  therefore,  as  the  pulse  rate  is  increased,  each  individual  laser  pulse 
will  necessarily  have  less  power.  It  is  possible  to  design  a  combination  of  flight  altitude 
and  pulse  rate  that  will  cause  the  return  signals  to  be  too  weak  to  trigger  the  receiver 
and  record  a  return.  This  limitation  is  especially  notable  in  vegetated  conditions,  where 
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the  success  of  the  survey  will  depend  on  recording  multiple  returns.  Multiple  returns 
have  less  power  since  they  reflect  only  part  of  the  laser  pulse.  In  practice,  vendors 
should  monitor  power  levels,  pulse  rates,  and  flight  altitude  in  order  to  maximize  the 
performance  of  the  system.  During  data  collection,  numerous  low-intensity  returns,  or 
failing  to  get  returns  over  less  reflective  surfaces  such  as  blacktop,  indicate  that  power 
should  be  increased,  altitude  lowered,  or  pulse  rate  reduced. 

1.3.2  Orthophotography 

The  orthophoto  system  consists  of  a  digital  camera,  with  a  clock  timer  to  record  the  time 
of  exposure.  The  time  record  allows  of  the  location  of  the  image  center  to  be  calculated 
using  data  from  GPS  and  INS. 

Since  the  mid-1990s,  camera  image  size  has  advanced  from  1,500  pixels  across  an 
image  to  approximately  4,500  pixels.  This  has  allowed  for  increased  flying  heights  and  a 
reduced  number  of  images  for  a  given  area,  with  consequent  cost  savings.  Cameras 
with  image  sizes  of  4,000  by  4,000  pixels  or  greater  are  often  favored  (at  the  time  of  this 
report),  since  the  width  of  the  digital  image  is  similar  to  the  typical  width  of  the  lidar 
swath. 
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2  LIDAR  AND  ORTHOPHOTOS  IN  MUNITIONS  MANAGEMENT 

Many  millions  of  acres  of  Department  of  Defense  (DoD)  lands  are  potentially 
contaminated  with  military  munitions  or  their  components.  On  the  majority  of  these  sites, 
munitions  are  concentrated  in  specific  ranges  and  training  areas,  while  the  remainder  of 
the  site  is  ordnance-free.  Locating  the  site  of  contamination  can  be  difficult,  in  part 
because  historical  records  are  often  incomplete  or  inaccurate.  The  cost  of  traditional 
surveys  using  magnetometry  and  electromagnetic  induction  (EMI)  can  be  very  high,  and 
this  has  driven  the  search  for  innovative  methods  to  reduce  costs. 

2.1  The  ESTCP  Wide  Area  Assessment  Pilot  Project 

Between  2005  and  2007,  the  Environmental  Security  Technology  Certification  Program 
(ESTCP)  conducted  a  pilot  program  to  test  the  effectiveness  of  a  multi-technology 
approach  to  unexploded  ordnance  (UXO)  WAA.  The  objective  of  WAA  is  to  “quickly  and 
cost  effectively  assess  100%  of  a  potentially  contaminated  site.  Beginning  with  historical 
records,  a  CSM  is  used  to  record  the  best  understanding  of  the  site.  The  WAA  is  a 
means  to  gather  a  preponderance  of  evidence  that  improves  our  understanding  of  the 
site  and  builds  confidence  in  the  conclusions.  A  suite  of  commercially  available 
technologies  provides  data  to: 

■  Identify  areas  of  concentrated  munitions  use 

■  Collect  information  that  will  support  decisions  on  areas  with  no  indication  of 
munitions  presence 

■  Collect  data  to  support  planning,  prioritization  and  contracting  when  a  site  must 
ultimately  be  cleaned  up”  (Nelson  et.  al.  2008) 

The  ESTCP  program  examined  the  use  of  lidar,  orthophotography,  helicopter 
magnetometry,  towed-array  magnetometry,  synthetic  aperture  radar,  hyperspectral 
sensing,  and  statistically-based  transect  design. 

At  the  demonstration  sites  where  lidar  and  orthophotos  were  tested,  the  results  were 
positive.  Lidar  and  orthophotos  were  used  to  locate  berms,  bombing  targets,  and 
individual  craters,  even  where  these  features  were  highly  eroded  and  could  not  be 
detected  by  ground-based  field  crews  (Figure  5).  The  location  of  these  features  was 
used  to  correct  the  initial  CSM,  to  refine  the  boundaries  of  munitions  response  sites 
(MRS)  and  to  support  and  help  direct  the  use  of  subsequent  technologies  that  directly 
detect  ordnance.  Results  from  lidar  and  orthophotos  provided  cross-validation  of 
magnetometry  and  EMI  data,  leading  to  higher  confidence  levels,  and  the  combination  of 
technologies  employed  in  the  pilot  program  was  found  to  be  cost-effective  (Nelson  et  al. 
2008). 


8 


Guidance  Document:  Using  Lidar  and  Orthophotography 

in  UXO  Wide  Area  Assessment 
Project  Number  07E-MM2-012/MM-0737 


Figure  5:  ESTCP  Demonstration  Sites,  Example  Munitions-Related  Features 


Kirtland  Air  Force  Base  Precision  Bombing  Range  ESTCP  demonstration  site,  bombing  targets  from  the  1940s 
visible  in  lidar-derived  surface  models.  These  targets  are  highly  eroded  and  cannot  be  seen  from  the  ground  or 
in  aerial  photos. 


Kirtland  site,  bombing  target  cross-hairs  visible  in 
10-cm  pixel  orthophoto. 


Victorville  Demolition  Bombing  Range  ESTCP 
demonstration  site,  crater  locations  outside  of  the 
original  dry  lake  bed  target  area. 


9 


Guidance  Document:  Using  Lidar  and  Orthophotography 
in  UXO  Wide  Area  Assessment 
Project  Number  07E-MM2-012/MM-0737 


Figure  5  continued 


Former  Camp  Beale  ESTCP  demonstration  site, 
bombing  target  (red  circle)  located  outside  of 
mapped  target  locations  based  on  historical  records. 


Pueblo  Precision  Bombing  Range  ESTCP 
demonstration  site,  crater  density  map  used  to  refine 
original  MRS  boundary. 


2.2  Advantages  of  Lidar  and  Orthophotos  at  Munitions  Management  Sites 

Using  lidar  and  orthophotos  offers  several  advantages  as  a  supplement  to  more 
traditional  approaches  to  site  investigation. 

■  Rate  of  coverage.  In  most  operational  settings,  data  collection  rates  of  5,000 
acres  per  day  or  higher  can  be  expected  for  lidar  and  orthophotos,  and  collection 
rates  of  up  to  20,000  acres  are  possible.  This  compares  favorably  to  typical 
collection  rates  of  around  500  acres  per  day  for  helicopter-based  magnetometry, 

5  to  20  acres  per  day  for  towed-array  magnetometry,  and  1  to  5  acres  per  day  for 
man-portable  magnetometry. 

■  Enhanced  planning  and  risk  assessment.  Because  they  can  cover  entire  sites 
relatively  quickly  and  at  lower  cost,  these  technologies  can  be  used  to  locate  and 
prioritize  appropriate  areas  for  use  of  more  costly  low-altitude  and  ground-based 
technologies. 

■  Ability  to  delineate  MRS  and  UXO-related  features.  Lidar  and 
orthophotography  can,  under  some  circumstances,  successfully  reveal  MRS  and 
UXO-related  surface  features  even  many  years  after  their  last  use. 

■  Cross-validation.  Lidar  and  orthophotos  can  cross-validate  the  results  of  other 
technologies,  leading  to  enhanced  confidence  in  results. 

■  Other  benefits.  Both  technologies  provide  highly  detailed  topographic  data  that 
can  be  integrated  into  a  facility’s  CADD  or  GIS  system  and  used  in  subsequent 
phases  of  site  investigation,  site  remediation,  and  range  management. 
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2.3  Limitations  of  Lidar  and  Orthophotos  at  Munitions  Management  Sites 

The  primary  limitation  of  lidar  and  orthophotos  is  that  neither  technology  can  directly 
detect  ordnance  or  UXO  components,  such  as  ferrous  scrap.  Consequently,  lidar  and 
orthophotos  are  not  substitutes  for  technologies  (such  as  magnetometry  or  EMI)  that 
directly  detect  ordnance.  Rather,  lidar  and  orthophotos  are  used  to  give  a  rapid 
assessment  of  potential  areas  of  ordnance  use,  based  on  ground  features  such  as  target 
objects,  crater  fields,  OB/OD  areas,  berms,  or  roads.  Areas  of  suspected  munitions  use 
can  then  be  prioritized  for  additional  survey.  However,  areas  appearing  to  be  free  of 
evidence  of  munitions  use  in  lidar  and  orthophotos  will  still  need  to  be  surveyed  with 
magnetometry  or  EMI  at  a  level  sufficient  to  demonstrate  that  they  are  actually 
ordnance-free. 

Lidar  and  orthophotos  are  best  used  as  the  first  step  in  site  investigation  to  quickly 
identify  areas  of  potential  contamination  and  other  areas  of  interest  and  to  help  direct  the 
application  of  subsequent  technologies.  Lidar  and  orthophotos  will  be  less  useful  on 
sites  that  have  been  fully  characterized  already  using  more  traditional  methods,  or  on 
small  sites  where  the  full  site  can  be  surveyed  quickly  using  ground  crews. 

An  additional  limitation  of  both  lidar  and  orthophotos  is  that  both  are  affected  by 
vegetation.  At  vegetated  sites,  orthophotos  will  show  only  the  tops  of  the  vegetation  and 
thus  are  of  limited  utility  in  characterizing  the  ground  surface.  Lidar  will  provide  useable 
surface  models  under  many  vegetation  conditions,  and  will  only  show  severe 
degradation  where  the  vegetation  is  particularly  dense.  However,  vegetation  will  lower 
the  density  of  lidar  returns  from  the  ground  surface  and  will  thus  lower  confidence  levels 
for  detecting  features  under  vegetation.  Vegetation  effects  are  discussed  in  more  detail 
in  Section  4.2. 
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3  ACQUIRING  LIDAR  AND  ORTHOPHOTO  DATA 
3.1  Pre-Mission  Planning 

Pre-mission  planning  is  an  essential  part  of  the  successful  use  of  lidar  and  orthophotos. 

Establishing  the  study  area  boundary.  The  lidar  vendor  must  be  provided  with  an 
accurate  study  area  boundary  in  order  to  plan  flight  lines  and  estimate  costs.  In  addition 
to  any  paper  maps  sent,  the  study  area  should  be  delivered  as  an  electronic  file  with 
spatial  coordinates.  ESRI3  shape  files  or  AutoCAD  drawing  files  with  spatial  coordinates 
are  the  most  common  formats.  Files  provided  for  planning  should  be  accompanied  by 
documentation  defining  the  sources  of  the  information,  and  any  specifications  associated 
with  the  data  so  this  information  can  be  carried  throughout  the  process. 

Establishing  survey  control  requirements.  The  project  survey  network  is  a  series  of 
points  established  with  survey-grade  GPS  techniques,  and  referenced  to  local  survey 
control.  The  number  of  control  points  to  be  established  depends  on  the  level  of 
validation  required,  with  the  understanding  that  error  may  not  be  distributed  equally 
throughout  the  site.  At  a  minimum,  validation  points  should  be  located  in  each  major 
terrain  type,  and  control  points  should  be  collected  at  an  accuracy  which  is  an  order  of 
magnitude  higher  than  the  lidar  points.  See  further  discussion  of  survey  control  in 
Section  3.2. 

Clarifying  projection  and  datum  requirements.  Lidar  data  will  be  integrated  with 
other  project  spatial  data,  and  therefore  should  be  delivered  in  the  projection  and  datum 
that  is  being  used  by  other  team  members.  On  large  projects,  it  is  a  reasonably 
common  problem  for  different  subcontractors  to  operate  with  data  in  different 
projections.  This  is  especially  problematic  in  the  case  of  lidar  because  the  data  sets  are 
very  large,  making  re-projection  time-consuming  and  difficult.  It  is  more  effective  to  give 
the  vendor  clear  direction  as  to  the  projection  and  datum  as  part  of  the  contract 
specifications. 

Flight  line  planning.  Flight  lines  are  planned  using  custom  software  to  achieve  planned 
flight  line  overlap  and  lidar  point  densities.  Flight  line  overlap  is  designed  to  allow  the 
lidar  signals  at  the  extreme  edge  of  the  swath  to  be  discarded.  Such  signals  are  less 
vertical  and  thus  are  more  likely  to  be  obstructed  from  reaching  the  ground  surface  by 
vegetation,  and  the  points  are  somewhat  more  widely  spaced  and  slightly  less  accurate. 
Flight  line  overlap  also  helps  to  prevent  data  gaps  when  the  aircraft  makes  minor  course 
deviations  due  to  wind,  a  common  occurrence  especially  with  helicopters.  Flight  lines 
should  be  designed  to  produce  a  relatively  symmetrical  distribution  of  laser  returns  on 
the  ground  surface,  which  assists  in  the  reliable  detection  of  small  objects. 

Establishing  data  density  requirements.  Selecting  appropriate  lidar  and  orthophoto 
data  densities  is  important  since  these  have  a  strong  impact  on  both  cost  and 
performance.  Density  specifications  for  both  technologies  should  be  adjusted  to  the 
specifics  of  each  site.  See  the  discussion  of  data  density  in  Section  4.1 . 

Field  logistics.  During  the  planning  phase,  arrangements  are  made  for  aircraft,  aircraft 
fuel,  and  a  hanger  in  which  to  install  the  equipment.  If  the  study  area  is  sufficiently  far 
from  the  nearest  airport,  it  may  be  necessary  to  establish  secondary  landing  areas,  GPS 


3  ESRI  is  the  producer  of  ArcGIS,  the  most  commonly-used  GIS  program  in  the  United  States. 
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base  stations,  or  fuel  sources.  Secondary  landing  areas  may  be  appropriate  for  helicopter 
surveys.  At  active  sites,  pilot  briefings  may  be  required  and  will  need  to  be  scheduled. 
During  the  planning  phase,  arrangements  also  are  made  for  housing  and  work  space  for 
the  data  collection  crew.  Site  access  problems  are  identified  and  resolved,  including 
ground  access  to  establish  survey  control  points  and  permission  to  enter  the  air  space  over 
the  site.  Contingencies  for  weather.  During  the  planning  phase,  site  weather  conditions  are 
assessed  and  the  mission  plan  adjusted  to  include  an  appropriate  contingency  for  weather 
delays.  Lidar  and  orthophotos  cannot  be  collected  in  wind,  rain  or  snow.  Choosing  the 
airframe.  Lidar  may  be  acquired  either  from  helicopters  or  small  fixed-wing  aircraft.  There 
is  generally  only  a  small  difference  between  the  two  airframes  in  cost  or  performance  and 
the  choice  can  usually  be  left  to  the  vendor.  In  practice,  some  performance  factors  include: 

•  Fixed-wing  aircraft  generally  fly  somewhat  faster  and  at  higher  altitudes  and 
therefore  achieve  somewhat  higher  data  collection  rates,  along  with  somewhat 
larger  orthophoto  pixel  sizes,  generally  12  to  20  cm.  If  10  cm  pixel  sizes  are 
required,  it  may  be  necessary  to  use  helicopters. 

•  Helicopters  can  sometimes  fly  under  low  cloud  cover  that  may  make  fixed-wing 
acquisition  difficult  or  impossible.  In  coastal  areas  with  frequent  cloud  cover,  using 
helicopters  may  be  advantageous  for  this  reason.  Helicopters  also  may  be 
advantageous  for  maintaining  a  fixed  altitude  in  mountainous  areas. 

•  Helicopters  make  turns  more  easily  and  can  follow  irregular  project  area  boundaries 
with  greater  efficiency. 

•  Helicopters  can  fly  lower  and  more  slowly  than  fixed-wing  aircraft,  thus  achieving 
higher  point  densities  without  flying  multiple  passes. 


A  second  consideration  in  choosing  an  airframe  is  the  use  of  local  versus  vendor-owned 
aircraft  and  pilots.  Some  vendors  own  and  operate  their  own  aircraft  and  provide  their  own 
pilots,  others  rent  local  aircraft.  The  choice  can  usually  be  left  to  the  vendor. 

•  Local  pilots  will  generally  be  more  familiar  with  local  weather  conditions,  airports, 
restricted  air  spaces,  and  flight  operational  controls.  This  can  be  an  advantage  in 
areas  with  unpredictable  or  difficult  weather  conditions,  or  where  restricted  air 
space  requires  advance  permission  and  coordination. 

•  Vendor-provided  pilots  will  generally  have  more  experience  at  collection  lidar  and 
orthophoto  data,  which  requires  staying  on  line,  at  speed  and  at  altitude  in  order  to 
maintain  data  quality  and  safety. 

•  Aircraft  owned  by  the  lidar  vendor  must  be  mobilized  to  the  project  site,  which  can 
be  time-consuming  and  lead  to  costly  delays  due  to  weather  and  border  crossings. 
In  contrast,  use  of  a  rented  aircraft  involves  shipping  only  the  sensor  equipment. 
Local  aircraft,  however,  can  be  difficult  to  rent  during  periods  of  strong  local 
demand,  and  must  be  verified  to  meet  the  technical  requirements  of  the  lidar 
system,  such  as  an  adequate  and  reliable  power  system. 


Project  schedule.  The  data  acquisition  schedule  is  based  on  expected  data  collection 
rates,  mobilization  requirements,  and  quality  control  procedures.  For  most  projects,  a 
collection  rate  of  5,000  acres  per  day  can  be  used  for  planning  purposes;  however, 
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collection  rates  up  to  20,000  acres  per  day  are  achievable,  particularly  with  fixed-wing 
aircraft.  Individual  sites  should  be  evaluated  for  site-specific  problems.  These  can 
include  site  boundaries  that  require  short  flight  lines  and  multiple  turns,  or  restricted  air 
space  where  flights  may  be  conducted  only  during  specified  hours.  For  all  projects,  an 
additional  day  should  be  planned  at  the  end  of  the  survey  for  re-acquiring  any  missed 
areas  or  areas  where  the  data  appear  to  be  problematic. 

Safety.  Lidar  and  orthophoto  data  collection  is  generally  governed  by  the  same  safety 
rules  and  considerations  as  any  small  aircraft  mission.  Local  aircraft  vendors  or  lidar 
vendors  who  operate  their  own  aircraft  should  provide  a  standard  safety  plan  as  part  of 
their  qualifications.  One  safety  problem  that  has  been  encountered  occasionally  is  data 
collection  that  occurs  near  an  existing  airport,  where  planned  flight  lines  may  conflict  with 
existing  air  traffic  patterns.  All  flight  lines  should  be  reviewed  carefully  with  local  airport 
managers  and  other  knowledgeable  sources  to  avoid  such  hazards.  For  collection  near 
existing  airports,  a  Notice  to  Airmen  may  be  issued  to  notify  other  pilots  of  the  activity  in 
the  area. 

Delivery  products.  During  mission  planning,  the  delivery  products  and  data  formats 
should  be  clearly  specified  and  agreed  upon.  Vendors  should  be  required  to  deliver  the 
full  lidar  data  set,  in  addition  to  any  derived  products  such  as  contour  lines  or  surface 
models.  The  choice  of  file  format  and  maximum  data  file  size  should  be  determined  by 
the  software  and  hardware  in  use  by  the  facility  receiving  the  data. 

Permitting  and  other  access  constraints.  Lidar  and  orthophoto  surveys  generally  do 
not  present  problems  with  permitting  and  site  access,  since  the  only  ground  access 
requirement  is  for  establishing  survey  control.  However,  access  to  local  air  space  at 
some  active  military  bases  can  occasionally  require  extensive  coordination  and  should 
be  investigated  well  in  advance.  For  international  projects,  the  IMU  is  US  military  dual 
use  technology,  and  international  use  requires  a  permit  pursuant  to  the  International 
Traffic  in  Arms  Regulations,  Code  of  Federal  Regulations  Sections  123.1  and  123.9(c). 
This  regulation  restricts  use  of  lidar  and  orthophotos  in  some  countries.  In  addition, 
foreign  countries  may  have  limitations  on  the  use  of  these  technologies  in  some 
sensitive  areas  (particularly  border  and  military  areas)  and  some  require  data  processing 
be  done  in-country  and  limit  outside  access  to  data.  For  international  projects,  sufficient 
time  must  be  allowed  for  permitting  and  shipping  requirements, generally  at  least  two 
weeks  to  ensure  that  the  IMU  will  arrive  at  the  destination. 

3.2  Field  Data  Acquisition 

Field  communication.  Communication  with  the  vendor  during  data  collection  is  critical 
to  the  success  of  the  survey.  During  data  collection,  site  conditions  often  force 
modification  of  mission  plans,  and  decisions  often  must  be  made  quickly.  Vendors 
should  be  in  daily  contact  with  the  Government  project  manager  or  other  representative 
throughout  the  field  effort,  and  all  decisions  should  be  reviewed  by  both  the  vendor  and 
the  Government  before  decisions  are  finalized.  The  Government  representative  must  be 
available  full-time  during  the  field  effort.  Daily  field  reports  are  generally  submitted  to 
document  the  day’s  activities  and  note  any  problems  encountered  or  resolved,  site 
visitors,  etc. 

Mobilization.  During  mobilization,  the  lidar  equipment  is  shipped  to  the  site,  installed  on 
the  aircraft  to  be  used,  and  tested.  Installation  typically  requires  less  than  one  day.  The 
data  collection  crew  also  establishes  a  work  area  where  data  can  be  processed  during 
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the  daily  quality  assurance/quality  control  (QA/QC)  review.  Fuel  is  placed  on  site  if 
needed. 

Establish  survey  control  points  and  other  calibration  items.  Just  prior  to  data 
collection,  control  points  are  established.  Primary  control  points  are  established  using 
static  survey  methods,  additional  surveyed  points  may  be  established  using  static  and/or 
kinematic  methods.  Fiberglass  targets  can  be  placed  with  the  control  points  for 
verification  of  horizontal  positional  accuracy  and  lidar-to-orthophoto  alignment.  Often, 
conditions  in  the  field  will  require  some  deviation  from  planned  control  point  locations. 
This  is  acceptable  as  long  as  an  adequate  network  of  control  points  is  maintained. 

It  is  often  desirable  to  establish  other  calibration  items  in  addition  to  control  points.  The 
most  common  are  vertical  control  structures  and  simulated  craters  at  a  variety  of  sizes. 
Detection  of  these  calibration  objects  can  be  used  in  a  manner  somewhat  analogous  to 
the  use  of  geophysical  prove-out  areas  in  traditional  geophysical  munitions 
investigations.  Current  best  practice  is  to  take  a  digital  photo  of  all  static  control  points 
and  other  calibration  objects,  and  to  survey  the  location  of  all  calibration  items. 

Sensor  calibration.  Before  data  acquisition  begins,  test  flights  are  conducted  over 
calibration  features  with  known  dimensions.  Calibration  flights  are  conducted  at  several 
headings.  On  return  to  the  office,  the  results  of  these  test  fights  are  used  to  establish 
correction  values  for  pitch,  roll,  and  yaw  so  that  the  features  remain  constant  at  all 
headings.  The  results  of  sensor  calibration  flights  should  be  recorded  as  part  of  the 
vendor’s  QA/QC  report.  Calibration  results  should  be  checked  daily.  Calibration 
coefficients  change  over  time  (on  the  scale  of  days)  and  their  suitability  should  be 
assessed  for  each  individual  data  set.  Long  deployments  (several  days  to  many  weeks) 
may  require  multiple  unique  sets  of  calibration  coefficients  to  account  for  small  yet 
consistent  drift  of  the  IMU. 

Data  acquisition  flights.  The  efficiency  of  the  data  acquisition  process  will  depend 
primarily  on  the  distance  to  the  nearest  fuel  source  and  the  percentage  of  the  flight  spent 
turning  vs.  acquiring  data.  Data  acquisition  also  can  be  affected  by  air  temperatures 
over  100°  F  during  the  summer  months,  which  can  affect  equipment  performance  and 
reduce  the  hours  of  operation.  During  flights,  the  operator  should  monitor  the  quality  of 
the  data  being  received  and  make  any  needed  corrections  to  operating  conditions  to 
assure  data  quality. 

In-field  QA/QC  review  and  re-flight.  All  vendors  have  software  with  which  the  field 
crew  can  examine  the  data,  and  daily  review  to  assess  data  quality  and  identify  any 
problems  is  a  standard  and  essential  part  of  field  activities.  Identifying  missed  areas  or 
data  quality  problems  allows  for  correction  in  the  field  prior  to  demobilization,  usually  by 
re-flying  missed  areas  or  areas  of  poor-quality  data.  Correcting  problems  of  this  kind 
while  the  crew  is  still  in  the  field  avoids  the  substantial  costs  and  delays  of  re¬ 
mobilization  and  re-acquisition. 

Documentation  of  the  field  activities  is  an  important  part  of  the  QA/QC  process. 
Documentation  should  include  site  photos,  calibration  records,  flight  logs  showing  time 
and  date  of  each  flight  line,  and  the  results  of  the  daily  QA/QC  review. 
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3.3  Post-Mission  Data  Processing 

Once  the  data  arrives  from  the  field,  the  vendor  performs  a  series  of  steps  to  produce 
lidar  points  and  orthophoto  images  that  are  ready  for  use  in  the  GIS,  CADD,  or 
geophysical  software  environment. 

3.3.1  Lidar 

Calibration.  On  return  to  the  office,  the  results  of  the  calibration  fights  are  used  to 
establish  calibration  values  for  pitch,  roll,  and  yaw  so  that  the  features  remain  constant 
at  all  headings.  These  correction  values  are  applied  either  to  the  entire  lidar  data  set,  or 
if  multiple  calibration  coefficients  are  developed,  each  will  be  applied  to  the  appropriate 
portion  of  the  lidar  data. 

Flight  line  integration.  Data  from  different  flight  lines  are  adjusted  to  achieve  the 
closest  possible  fit  to  each  other  and  to  the  project  control  points.  This  calibration 
process  is  used  to  correct,  to  the  extent  possible,  minor  positional  error  from  one  flight 
line  to  the  next,  and  to  detect  any  major  positional  errors.  This  calibration  is  essential  to 
assure  that  lidar  points  will  be  positionally  accurate  throughout  the  project  area.  Small 
discrepancies  will  always  remain  between  the  data  from  one  flight  line  to  the  next;  this  is 
acceptable  as  long  as  the  discrepancies  fall  within  the  contract  specifications.  Large 
discrepancies  between  flight  lines  that  cannot  be  corrected  may  indicate  serious 
equipment  or  data  collection  problems  and  may  require  re-acquisition. 

QA/QC  review.  As  part  of  the  calibration  procedure,  the  vendor  should  perform  basic 
QA/QC  review  of  the  alignment  of  the  lidar  data  to  the  control  points.  This  review  forms 
the  basis  for  validating  the  location  of  the  lidar  data.  If  vertical  control  structures  and 
simulated  craters  have  been  established,  detecting  these  items  should  be  part  of  this 
QA/QC  review. 

Lidar  point  classification.  Once  the  lidar  data  are  calibrated,  the  ground  surface  is 
derived  from  the  entire  lidar  point  data  set,  and  remaining  points  are  classified  as 
vegetation,  buildings,  and  other  objects,  depending  on  the  needs  of  the  project. 
Classification  is  done  through  automated  algorithms  followed  by  inspection  and  editing. 
Inspection  and  editing  by  qualified  analysts  is  essential  to  creating  a  quality  ground 
surface  model. 

Point  classification  methods  can  affect  the  density  of  ground  points  and  thus  the 
resolution  of  the  ground  surface  model.  Typical  software  settings  tend  to  favor  creation 
of  smooth  ground  surface  models,  and  can  result  in  some  ground  points  being 
misclassified  as  non-ground  in  order  to  keep  the  surface  model  smooth.  Point 
classification  methods  can  be  adjusted  to  add  more  points  to  the  ground  surface  model, 
and  work  at  the  ESTCP  demonstration  sites  has  shown  that  this  may  increase  the 
resolution  of  the  surface  model  and  somewhat  increase  detection  of  small  objects  (URS 
2009a).  When  the  objective  of  the  survey  is  to  detect  objects  such  as  small  craters,  it  is 
important  to  discuss  point  classification  methods  with  the  vendor  in  order  to  maximize 
the  possibility  of  detection. 

Data  delivery.  Once  the  lidar  points  have  been  created,  they  are  delivered  by  the 
vendor,  along  with  any  additional  products  such  as  surface  models.  Data  delivery 
requires  several  decisions: 

■  Format.  Lidar  points  can  be  delivered  in  several  different  formats.  The  most 
common  are  ASCII  text  files  and  the  LAS  file  format.  LAS  is  a  binary  format  that 
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was  specifically  designed  for  lidar  data.  LAS  files  are  considerably  smaller  than 
equivalent  ASCII  files,  and  LAS  files  can  be  viewed  directly  in  many  software 
products  without  conversion.  ArcGIS  requires  a  third-party  extension  to  directly 
view  and  process  LAS  files,  or  they  can  be  converted  to  ESRI  shapefiles  using 
free  tools  that  can  be  found  on  the  internet.  In  the  ArcGIS  environment,  LAS  files 
also  can  be  converted  to  a  special  ArcGIS  feature  class  type  called  a  multipoint 
feature  class.  The  multipoint  feature  class,  however,  does  not  allow  direct 
access  to  the  file  attributes,  thus  preventing  any  kind  of  analysis,  manipulation,  or 
reclassification  based  on  attribute  values  of  the  lidar  points.  ASCII  text  files, 
usually  in  space,  tab  or  comma-separated  format  have  the  advantage  of  being 
universally  accepted.  The  files  are  much  larger,  although  these  files  compress 
well  into  much  smaller  archive  formats  such  as  ZIP,  RAR,  LZH,  GZ,  etc. 

■  Delivery.  Lidar  data  should  be  delivered  using  a  DVD,  flash  drive,  or  portable 
hard  drive,  depending  on  the  size  of  the  data  set.  Ftp  is  a  potential  method  of 
data  transfer,  but  often  leads  to  problems.  Ftp  transfer  is  generally  very  slow 
given  the  size  of  lidar  files,  and  it  is  possible  for  data  files  to  become  corrupted 
during  transfer. 

■  Data  tiles.  Lidar  and  orthophoto  file  sizes  can  be  very  large,  and  data  is 
generally  delivered  in  tiles.  It  is  important  that  the  lidar  vendor  deliver  the  data  in 
tile  sizes,  and  resulting  data  files,  that  can  be  managed  efficiently.  File  sizes 
present  a  tradeoff:  larger  tiles  result  in  larger  data  files,  which  are  harder  to  open, 
examine,  process,  copy,  or  move,  but  large  numbers  of  smaller  files  can  be 
confusing  and  difficult  to  manage.  File  sizes  should  be  determined  by  the 
software  and  hardware  capabilities  of  the  receiving  installation. 

■  Overlapping  vs.  non-overlapping  blocks.  Lidar  can  be  delivered  in  blocks  with 
or  without  overlap.  Non-overlapping  blocks  will  generally  result  in  data  gaps 
between  blocks  if  they  are  processed  separately  and  recombined  in  the  analysis 
stage  (URS  2007).  If  processing  in  separate  blocks  is  desired,  all  vendors  can 
deliver  overlapping  blocks  upon  request,  but  this  is  not  always  the  default  so  this 
requirement  should  be  included  in  the  contract  specifications.  As  an  alternative, 

it  may  be  possible  to  avoid  data  gaps  between  blocks  by  using  the  ESRI 
multipoint  feature  class,  or  a  software  package  other  than  ArcGIS,  to  load  all  of  a 
project  area’s  lidar  data  into  a  single  file  for  creating  surface  models. 

■  Lidar  point  attributes.  Lidar  points  should  include  several  attributes  in  addition 
to  the  x,  y,  and  z  values.  These  include  values  for: 

-  Class:  The  vendor’s  classification  of  the  point  as  a  ground  or  non-ground 
return.4 

-  Intensity:  The  strength  of  the  signal  returned. 

-  Flight  line:  The  code  for  an  individual  flight  line.  It  is  useful  to  view  data 
from  individual  flight  lines,  especially  as  a  means  to  diagnose  problems  with 
the  data. 


4  Vendors  can  use  more  complex  classification  schemes  if  desired,  including  classification  as  vegetation, 
buildings,  or  power  lines. 
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-  Collection  date  and  time:  These  are  values  for  GPS  week  and  GPS  time 
that  are  collected  automatically  as  part  of  system  operations. 

The  vendor  should  provide  documentation  of  all  attribute  codes,  such  as  the  values  to 
indicate  ground  and  non-ground  returns  and  the  format  used  for  the  collection  date  or 
flight  line  number. 

3.3.2  Orthophotos 

Once  the  individual  digital  images  are  collected,  vendors  create  orthophotos  by  building 
a  mosaic  of  the  individual  images,  transforming  the  consolidated  image  to  the  delivery 
datum  and  projection,  orthorectification  using  the  lidar  data,  color  balancing,  and 
trimming  to  the  delivery  tiles.  While  some  of  these  processes  can  be  automated, 
creating  high  quality  orthophoto  images  requires  a  large  amount  of  operator  time, 
especially  for  creating  the  image  mosaic.  Consequently,  orthophotos  with  smaller  pixel 
size  are  more  expensive  and  time  consuming  to  create,  since  they  involve  manipulating 
larger  numbers  of  smaller  images. 

Orthophotos  should  be  delivered  in  tiles  sized  to  keep  the  files  at  a  useable  size.  This 
size  limit  should  be  determined  based  on  the  software  in  which  the  data  will  be  used. 

3.3.3  Post-Mission  Quality  Control  Report 

At  the  conclusion  of  data  processing,  the  vendor  performs  a  series  of  QC  checks  based 
on  the  contract  specifications,  and  these  are  summarized  in  a  QC  report.  This  report  is 
typically  submitted  with  the  data.  The  QC  report  should  include,  at  a  minimum: 

■  Accuracy  of  the  control  points 

■  Position  dilution  of  precision  during  data  collection 

■  Data  collection  specifications 

■  Root  mean  square  values  for  orthophoto  triangulation 

■  Locations  of  control  points  and  GPS  base  stations 

■  Site  photos  of  the  data  collection  process  and  control  points 

■  Equipment  calibration  results 

■  Achieved  swath  width 

■  Achieved  swath  overlap 

■  Achieved  point  density 

■  Achieved  flight  altitudes 

■  Achieved  flight  lines  (as  CADD  or  shape  files) 

■  Lidar  vertical  positional  accuracy  compared  to  controls 

■  Lidar  horizontal  positional  accuracy  compared  to  controls 

■  Orthophoto  horizontal  positional  accuracy  compared  to  controls 

■  Orthophoto  positional  accuracy  compared  to  lidar 

■  Results  of  lidar  data  integration  between  flight  lines 
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3.4  Receiving  and  Working  with  Lidar  and  Orthophoto  Data 

3.4.1  Data  Management 

Lidar  and  orthophoto  data  sets  can  be  very  large,  and  managing  these  large  data  sets 
requires  several  important  considerations. 

File  naming  system.  Vendors  typically  deliver  lidar  point  data  and  orthophotos  as  a 
series  of  tiles  or  blocks  sized  to  keep  the  file  size  manageable.  A  large  site  can  require 
several  hundred  data  blocks,  and  the  difficulty  of  keeping  these  data  sets  organized  for  a 
large  site  should  not  be  underestimated.  Having  a  well-defined  file  naming  system  is 
essential.  Consistent  file  naming  also  is  required  in  the  use  of  automated  scripts  for 
batch  processing. 

Data  storage.  The  large  data  sets  produced  by  lidar  and  orthophotos  make  backup  and 
storage  an  ongoing  problem  for  many  users.  These  data  sets  can  sometimes  be  too 
large  to  back  up  using  an  organization’s  existing  tape  backup  system,  and  some  form  of 
independent  backup  and  storage  must  be  maintained.  The  optimal  solution  is  generally 
a  dedicated  server  with  tape  backup  of  adequate  size.  Where  this  is  too  expensive,  an 
alternative  is  to  use  a  small,  portable  “redundant  array  of  independent  disks”  (RAID) 
device,  which  are  available  at  relatively  low  cost.  RAID  devices  are  generally  acceptable 
as  storage;  however,  they  sometimes  present  problems  of  data  corruption  if  they  are 
used  for  data  processing.  Processing  should  be  performed  on  the  local  machine  where 
possible  and  copied  to  the  RAID  device  for  storage. 

Additional  files.  Because  the  data  are  delivered  in  blocks,  the  vendor  must  produce 
and  deliver  a  block  index  file  for  each  data  set.  Other  required  files  include  the  surveyed 
locations  of  all  control  points  and  calibration  structures,  line  files  for  all  achieved  flight 
lines,  and  the  QA/QC  report.  The  format  of  these  ancillary  files  should  be  compatible 
with  the  software  that  will  be  used  to  view  and  analyze  the  data. 

G/S  products.  Although  the  initial  data  sets  are  large,  the  final  GIS  products  from  lidar 
such  as  point  files  and  digital  elevation  models  (DEMs)  are  smaller,  and  are  within  the 
capability  of  normal  GIS  programs  to  process,  analyze,  and  display.  This  may  not  be  the 
case  for  CADD  and  geophysical  programs,  and  exporting  files  from  the  GIS  to  these 
software  environments  requires  additional  planning  and  may  require  clipping  the  data  to 
smaller  tiles. 

3.4.2  Creating  GIS  Products  from  Lidar  Points 

Lidar  data  arrives  from  the  vendor  as  a  table  of  values  in  one  of  several  text  file  formats 
(Figure  6).  These  data  must  be  converted  to  a  useable  format,  generally  in  a  GIS 
program,  before  they  can  be  reviewed  and  analyzed.  Products  to  be  created  include 
point  files  based  on  the  individual  lidar  points,  models  of  the  ground  surface  (DEMs),  and 
digital  terrain  models  (DTMs)  based  on  all  lidar  returns,  along  with  hill  shaded  views 
created  from  both.  The  lidar  vendor  will  provide  these  products  along  with  the  point  data 
if  requested;  however,  where  the  receiving  facility  has  GIS  capabilities  these  products 
may  be  produced  in-house.  Surface  models,  hillshades,  and  contour  lines  can  be 
created  and  displayed  in  many  different  ways,  each  of  which  will  affect  the  utility  of  the 
products.  Commonly,  several  different  approaches  will  be  tested,  and  different  products 
will  be  created  for  specific  parts  of  the  analysis.  The  advantage  of  creating  these 
products  in-house  is  that  these  choices  can  be  made  explicitly,  and  different  approaches 
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can  be  tested  more  easily,  assuming  that  the  appropriate  equipment  and  operators  are 
available. 


ESRI’s  ArcGIS  software  is  the  product  that  facilities  most  commonly  use  for  creating  and 
analyzing  lidar  data,  and  it  will  function  adequately  for  most  tasks.  Basic  GIS  products 
are  created  using  standard  GIS  data  processing  methods.  In  the  ESRI  GIS 
environment,  these  processes  can  be  automated  using  scripts  written  in  AML,  Python,  or 
Visual  Basic.  Once  basic  GIS  products  are  created  from  the  lidar  data,  products 
appropriate  to  other  software  packages,  such  as  CADD  or  geophysical  software,  can  be 
exported  at  will. 


Other  software  tools  are  available  for  more  specialized  analysis  tasks,  and  in  some 
cases  these  may  offer  capabilities  that  ArcGIS  does  not  provide. 


Figure  6:  Lidar  System  Output 
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|GPSWK, GPSTIME " XCOORD , YCOORD , ELEVATION, RETURNS , RETURNNUM, ANGLE , INTENS ITY, CLASS 

1261,333751.08851,7625230.53, 626267. 68, 194.94,269.84, 14.08,1 
1261,333751.08856,7625234.73, 626269.07, 194.55,269.44, 14.01, 1 
12  61,333751.08861,7  625240.58,  62  6270.81,  198. 17 , 2 73 . 07 ,  13 . 93 , 1 

1261.333751. 10891.7625219.50. 626268.76. 193 .36.268.25. 14.22.1 
12  61,333751. 10896,7625224.83, 626270.46,  194. 12 , 2 69 . 0 1 ,  14 . 14 ,  1 
1261,333751. 10901,7625231.45, 62 62 72 . 53 , 19 6 . 1 6 , 2 7 1 . 0 6 , 14 . 04 , 1 
1261,333751. 10906,7625236. 13, 626273 .93, 199.07,273 .96, 13 .97, 1 

1261.333751. 10911.7625240.39. 62 62 75 . 4 6 . 19 6 . 2 6 . 2 7 1 . 1 6 . 13 . 89 . 1 

1261.333751. 12927.7625204.28. 626268.47. 192 .59.267.49.14.44.1 
1261,333751. 12931,7625209.36, 626270. 12, 193 . 02 , 2 67 . 92 , 14 . 3 6 , 1 

1261.333751. 12936.7625214.94. 626271.86. 194.75.269. 64.14.27.1 

1261.333751. 12941.7625219.79. 626273 .47. 194.27.269. 16. 14. 19. 1 
12  61,333751. 12946,7  625224.98,  62  6275. 13,  195. 17 , 2 70 . 0 6 ,  14 . 1 1 ,  1 
1261,333751. 12951,7625230.22, 626276.82, 195. 64,270.53,14.03,1 

1261.333751. 12955.7625235.82. 62 62 78 . 60 . 19 6 . 92 . 2 7 1 . 8 1 . 13 . 94 . 1 

1261.333751. 12960.7625240.94. 626280.24. 197. 61. 2 72. 51. 13. 86.1 
1261,333751. 14967,7625194. 6 1 , 62 62 70 . 0 6 , 192 .01,266.91, 14.56, 1 

1261.333751. 14971.7625200.36. 626271.92. 192 .40.267.30.14.47.1 
1261,333751. 14976,7625205.09, 626273 .43, 193 .40,268.29,14.40,1 

1261.333751. 14981.7625210.96. 626275.34. 193 . 65 . 2 68 . 55 . 14 . 3 1 . 1 

1261.333751. 14986.7625215.27. 626276.72. 194.35.269.25. 14.24.1 
12  61,333751. 14991,7625220.70,  62 62 78 . 4 1 ,  19 6 . 2 2 , 2 7 1 . 12 ,  14 . 1 6 ,  1 
1261,333751. 14995,7625226. 64 , 62 62 80 . 2 6 , 198. 15,273.04,14.07,1 

1261.333751. 15000.7625231. 13 . 62 62 8 1 . 79 . 19 6 . 9 6 . 2 7 1 . 85 . 13 . 99 . 1 


Sample  lidar  point  locations  in  ASCII  comma-delimited  format.  Data  fields  are  included  for  GPS 
week,  GPS  time,  x,  y,  and  z  coordinates,  number  of  returns  for  this  pulse,  return  number  for  this 
return,  scan  angle,  intensity  value,  and  classification  code  for  ground  vs.  non-ground  return. 


Creating  point  files .  Point  files  are  the  first  product  to  be  created  from  the  lidar 
positional  locations.  All  of  the  fields  in  the  original  text  file  can  be  used  in  subsequent 
analysis,  and  should  be  kept  as  attributes  of  the  resulting  points.  For  instance,  retaining 
the  flight  line  number  as  an  attribute  of  the  lidar  points  allows  flight  line  overlap  to  be 
inspected  (Figure  7).  The  lidar  points  also  can  be  examined  to  show  variations  in  data 
density,  as  discussed  in  Section  4.1. 

Creating  surface  models .  The  most  common  products  created  from  lidar  data  are 
surface  models,  either  of  the  ground  surface  alone  (DEMs)  or  of  the  ground  surface  plus 
vegetation,  buildings  and  other  features  (DSMs).  As  discussed  in  Section  4.3.5,  surface 
models  can  be  created  in  a  variety  of  ways  and  these  methods  should  be  adapted  to  the 
needs  of  each  survey. 
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Creating  contour  lines.  Contour  lines  also  are  a  typical  product  of  lidar  data,  and 
contour  line  creation  introduces  some  important  technical  choices  (Figure  8).  Users  are 
often  surprised  that  lidar-based  contours  look  highly  irregular  and  jagged,  rather  than  the 
smooth  contours  that  are  usually  found  on  topographic  maps.  This  effect  is  a  result  of 
the  detailed  ground  surface  being  interpreted  through  computer  methods  rather  than  by 
human  operators,  as  in  photogrammetry.  Smooth  contours  can  be  created  from  lidar 
data;  however,  these  should  be  approached  with  some  caution,  and  adjusted  to  the 
particular  project  needs.  Caution  is  required  because  computer-based  methods  for 
creating  smooth  contour  lines  can  sometimes  remove  small  ground  features  (such  as 
craters  or  fault  scarps)  that  may  be  important  for  the  project. 


Figure  7:  Lidar  Flight  Line  Overlap 


Lidar  points  from  overlapping  flight  lines.  The  aircraft  traveled  east  to  west  (right  to  left).  The  impact  of  wind 
conditions  can  be  seen  at  the  edges  of  the  flight  lines. 


GIS  or  CADD  programs  can  create  contour  lines  at  any  interval.  However,  contour  lines 
cannot  be  certified  as  accurate  unless  the  underlying  data  meets  specific  requirements. 
As  a  general  rule,  creating  smaller  contour  line  intervals  requires  lidar  data  that  are 
denser  and  more  accurate. 


One  standard  source  for  guidance  on  accuracy  standards  for  contour  lines  and  other 
elevation  data  is  the  American  Society  for  Photogrammetry  and  Remote  Sensing 
(ASPRS)  publication  ASPRS  Guidelines:  Vertical  Accuracy  Reporting  for  Lidar  Data 
(ASPRS  2004).  In  terms  of  setting  accuracy  standards  for  contour  lines,  ASPRS  points 
out  that  contour  lines  are  produced  through  a  series  of  technical  steps,  each  of  which 
has  the  potential  to  introduce  error.  Consequently,  contour  line  accuracy  should  be 
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specified  for  the  contours  themselves,  and  should  not  be  confused  with  accuracy  of  the 
lidar  points.  The  guidelines  conclude  that  “Specifying  accuracy  of  the  final  product(s) 
requires  the  data  producer  to  ensure  that  error  is  kept  within  necessary  limits  during  all 
production  steps.” 


Figure  8:  Lidar-Based  Contour  Lines 


Former  Camp  Beale  ESTCP  demonstration  site,  ground  surface  model  and  original,  contour  lines  without  smoothing. 


Former  Camp  Beale  ESTCP  demonstration  site.  Contour  lines  increasingly  smoothed. 


Contour  line  accuracy  is  sometimes  certified  in  reference  to  the  National  Map  Accuracy 
Standards  (NMAS)  or  equivalent  (NDEP  2004).  Often,  contour  line  accuracy  is  required 
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to  be  certified  by  licensed  professionals  such  as  surveyors  or  photogrammetrists.  The 
NMAS,  referring  to  vertical  accuracy,  states: 

“Vertical  accuracy,  as  applied  to  contour  maps  on  all  publication  scales, 
shall  be  such  that  not  more  than  10  percent  of  the  elevations  tested  shall 
be  in  error  more  than  one-half  the  contour  interval.  In  checking  elevations 
taken  from  the  map,  the  apparent  vertical  error  may  be  decreased  by 
assuming  a  horizontal  displacement  within  the  permissible  horizontal 
error  for  a  map  of  that  scale.  ” 

At  least  one  standard  reference  notes  that  the  NMAS  was  developed  before  the  advent 
of  DEMs,  and  recommends  that  DEMs  generally  should  not  be  certified  to  NMAS 
standards  (ASPRS  2001).  This  could  apply  to  contour  lines  created  from  lidar  data, 
since  the  contour  lines  would  be  created  from  the  lidar-based  DEM. 

An  important  consideration  in  creating  contour  lines  is  that  vertical  error  will  be  lower  in 
flat,  open  areas  and  higher  in  rugged  terrain  or  areas  of  greater  vegetation  cover. 
Consequently,  in  applications  where  contour  line  accuracy  is  critical,  it  may  be 
appropriate  to  require  separate  reporting  of  lidar  accuracies  for  different  ground  cover 
categories.  This  would  require  placement  of  appropriate  survey  controls  in  each  ground 
cover  type. 

Vendors  contacted  reported  that  they  regularly  produced  2-foot  contour  lines  that  met 
ASPRS  standards,  and  occasionally  received  requests  for  1-foot  contour  lines.  They  felt 
that  creation  of  1-foot  contour  lines  required  a  point  density  of  at  least  two  to  three  points 
per  square  meter,  with  five  to  six  points  being  better,  and  with  control  points  distributed 
through  all  ground  cover  types.  Most  vendors  further  reported  that  they  would  not  create 
1-foot  contour  lines  for  steep  or  very  uneven  ground  surfaces,  where  horizontal  error  in 
the  lidar  data  would  result  in  unacceptable  vertical  error.  At  the  time  of  this  report,  a 
small  number  of  vendors  would  produce  contour  lines  at  0.5-foot  intervals;  however, 
such  surveys  require  extremely  high-accuracy  equipment,  very  low  flight  elevations,  and 
very  high  point  densities. 

3.4.3  Independent  Quality  Control 

Once  basic  GIS  products  have  been  created,  the  lidar  data  can  be  inspected  for  data 
quality  problems.  The  lidar  vendor’s  QC  report  should  not  substitute  for  this  independent 
review  of  the  quality  of  the  lidar  data.  This  section  reviews  steps  in  the  inspection 
process. 

Inspecting  for  artifacts.  As  surface  models  are  created,  they  should  be  inspected  for 
both  small  and  large  artifacts.  “Small”  artifacts  are  defined  as  those  that  are  within  the 
stated  horizontal  and  vertical  accuracy  of  the  data.  These  are  generally  the  result  of 
precision  error  as  described  in  Section  4.3.3.  Small  artifacts  of  this  type  cannot  be 
eliminated  entirely,  and  do  not  affect  the  level  of  analysis  needed  for  most  projects. 

Large  artifacts  are  those  that  exceed  the  data  accuracy  specification,  and  these  can 
indicate  improper  data  calibration,  editing,  or  processing.  Errors  of  this  type  can  require 
re-processing  or  even  re-acquisition  to  correct. 

Verifying  point  density.  Point  density  is  verified  by  calculating  the  total  number  of 
points  collected  per  square  meter  for  the  project  area  as  a  whole.  In  practice,  lidar 
density  varies  considerably  with  such  factors  as  flight  line  overlap,  as  discussed  in 
Section  4.1.3.  Overall  point  density  should  therefore  be  understood  as  a  measure  of  the 
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proper  functioning  of  the  equipment  and  compliance  with  general  contract  specifications. 
In  some  applications,  it  may  be  appropriate  to  specify  minimum  acceptable  point 
densities  for  smaller  areas  of  a  specified  size  or  for  each  major  terrain  type,  as 
discussed  in  Section  4.7. 

Verifying  positional  accuracy.  Vertical  and  horizontal  accuracy  are  verified 
separately.  Vertical  accuracy  is  verified  by  comparing  the  surveyed  elevation  of  the 
control  points  to  the  elevation  of  the  lidar  data.  Horizontal  accuracy  is  verified  by 
comparing  the  horizontal  location  of  a  surveyed  object  that  can  also  be  detected  in  the 
lidar  data.  Points  used  could  include  the  survey  targets,  or  surveyed  points  can  be 
placed  at  the  corners  of  buildings,  paved  surfaces,  or  other  natural  or  cultural  objects 
that  will  be  clearly  visible  in  the  lidar  elevation  or  intensity  data.  Lidar-to-orthophoto 
alignment  is  verified  by  comparing  the  surveyed  control  point  locations  to  their  locations 
in  the  orthophotos. 

3.4.4  Identifying  and  Analyzing  Ground  Features 

At  munitions  sites,  most  analysis  is  aimed  at  detecting  and  interpreting  ground  features, 
and  is  performed  by  examining  surface  models.  In  some  cases,  it  may  be  useful  to  view 
the  lidar  points,  in  either  two  or  three  dimensions. 

Identifying  and  evaluating  ground  features.  Generally,  personnel  with  experience  in 
munitions  sites  identify  ground  features  by  visually  inspecting  the  surface  models  and 
orthophotos.  Some  features  are  easily  identified  as  munitions-related.  These  include 
bombing  targets,  crater  fields,  and  OB/OD  areas  (Figure  9). 

Figure  9  Unambiguous  Munitions-Related  Ground  Features 


Other  ground  features  are  ambiguous,  and  their  presence  is  an  indication  that  further 
investigation  is  required  using  technologies  such  as  magnetometry  or  EMI.  Ambiguous 
features  include  individual  crater-shaped  objects  that  may  result  from  ordnance  use, 
other  human  activities,  or  natural  ground  variations  (Figure  10). 
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Figure  10:  Ambiguous  Ground  Features 


Former  Camp  Beale  ESTCP  demonstration  site.  Ground  feature  with  negative  magnetic  response. 


Former  Camp  Beale  ESTCP  demonstration  site.  Ground  feature  with  positive  magnetic  response. 

In-office  evaluation  of  potential  ground  features  is  typically  conducted  in  several  steps. 
First,  the  entire  set  of  potential  munitions-related  features  is  identified,  either  by  GIS 
analysts  or  using  automated  feature  extraction  methods.  The  aim  of  this  first  step  is  to 
identify  all  potential  man-made  features,  being  as  inclusive  as  possible. 

Second,  potential  features  are  compared  to  the  historic  information  in  the  CSM,  Archive 
Search  Report,  or  other  available  documents.  Staff  familiar  with  the  type  of  ordnance 
used  on  the  site  should  contribute  to  this  comparison. 

Finally,  the  individual  features  are  examined  by  staff  who  are  familiar  with  type  or 
ordnance  uses  on  the  site  and  the  size  and  shape  of  the  expected  features,  using  the 
orthophoto  and  lidar  data.  The  aim  is  to  identify  those  features  that  are  most  clearly 
related  to  ordnance  use,  those  that  are  clearly  not  ordnance-related,  and  those  that 
require  further  investigation. 

The  results  of  the  in-office  evaluation  can  be  used  to  correct  or  confirm  the  historic  data, 
to  modify  the  boundaries  of  MRS,  and  to  identify  ambiguous  features  for  further  field 
verification.  Once  complete,  the  evaluation  is  used  to  focus  and  prioritize  the  following 
stages  of  the  site  investigation. 
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4  Factors  Affecting  Performance  of  Lidar  and  Orthophotos 

The  performance  of  lidar  and  orthophotos  is  affected  by  a  variety  of  site  conditions  and 
processing  choices.  These  will  affect  both  the  expectations  for  the  technologies  and  the 
appropriate  contract  specifications  for  each  site. 

4.1  Data  Density  Effects 

An  important  decision  in  acquiring  lidar  and  orthophoto  data  is  the  density  of  the  data  to 
be  collected.  For  both  technologies,  data  density  directly  affects  both  the  detection 
ability  of  the  technology  and  the  cost  of  collecting  and  processing  the  data.  Orthophoto 
data  density  is  expressed  as  the  size  of  the  orthophoto  pixels.  Lidar  data  density  is 
expressed  as  either  the  average  number  of  points  per  square  meter  or  the  average 
spacing  between  lidar  points.  Unlike  orthophoto  pixel  size,  lidar  data  density  can  vary 
considerably  over  the  ground  surface.  In  areas  of  lower  lidar  density,  confidence  in 
detection  will  be  lower  for  small  ground  features. 

4.1.1  Orthophoto  Pixel  Size 

The  size  of  the  orthophoto  pixels  has  a  strong  effect  on  feature  detection  capability. 
Figure  1 1  compares  10  cm  and  20  cm  pixel  orthophotos  showing  simulated  craters  at 
the  Kirtland  Air  Force  Base  ESTCP  demonstration  site.  These  small  features  are  much 
more  clearly  visible  in  the  10-cm  image.  The  1.0  m  simulated  craters  are  visible  in  the 
10-cm  pixel  image  on  the  left.  The  0.3-m  craters  are  visible,  but  can  only  be 
distinguished  from  the  surrounding  vegetation  due  to  the  regularity  of  their  pattern. 
Isolated  ground  features  of  this  size  would  not  be  distinguishable. 

Figure  11:  Orthophotos  with  Simulated  Craters  -  10  cm  and  20  cm  pixels 


Kirtland  Air  Force  Base  ESTCP  demonstration  site,  10  cm  (left)  and  20  cm  (right)  pixel  orthophoto  with 
simulated  craters  at  0.3,  1.0,  and  1.5  m  diameters. 

The  advantage  of  smaller  pixel  sizes  is  not  limited  to  small  objects.  At  the  Former  Camp 
Beale  ESTCP  demonstration  site,  a  1 ,000-foot  diameter  bull’s-eye  target  was  visible  in 
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the  10  cm  orthophotos  acquired  for  the  project,  but  was  not  visible  in  a  previously- 
available  30  cm  pixel  orthophoto  (Figure  12). 

Figure  12:  Orthophotos  with  Bombing  Target  -10  m  and  30  cm  pixels 


Former  Camp  Beale  ESTCP  demonstration  site,  bull’s-eye  aiming  target:  10  cm  (left)  and  30  cm  (right)  pixel 
orthophotos.  The  target  is  visible  in  the  10  cm  pixel  orthophoto  but  not  in  the  30  cm  pixel  version. 

4.1.2  Lidar  Data  Density  and  Feature  Detection  Limits 

Work  at  the  ESTCP  demonstration  sites  showed  that  densities  of  four  points  per  square 
meter  and  higher  were  sufficient  to  characterize  the  simulated  craters  at  the  1 .0  m  and 
1 .5  m  sizes,  at  least  in  open  areas.  Higher  point  densities  tested  showed  ground 
features  with  more  clarity,  but  did  not  reveal  more  features  (URS  2007  and  2008).  For 
most  munitions  investigations,  four  to  five  points  per  square  meter  should  be  considered 
as  a  minimum  recommended  point  density,  with  higher  density  desirable  for  more 
vegetated  sites.  However,  simulated  craters  at  0.3-m  (1-foot)  diameter  were  not 
detected  reliably  at  any  of  the  lidar  densities  tested  (Figure  13)  (URS  2007  and  2008). 
Even  at  the  highest  data  densities  tested,  the  lines  of  lidar  points  could  miss  the  0.3-m 
diameter  craters.  At  the  few  areas  where  the  0.3-m  craters  were  detected,  they  were 
very  hard  to  distinguish  from  the  normal  variations  in  the  ground  surface,  and  probably 
would  not  have  been  noted  if  their  locations  had  not  been  surveyed  previously. 

4.1.3  Variations  in  Lidar  Data  Density 

Lidar  vendors  report  lidar  data  density  as  the  average  number  of  points  per  square 
meter,  or  the  average  spacing  between  points,  based  on  the  entire  site.  However,  in 
practice,  lidar  point  density  varies  considerably  over  the  site,  based  on  factors  such  as 
flight  line  overlap  and  wind  conditions  that  cause  the  aircraft  to  pitch,  roll,  and  yaw. 
Variation  in  overall  lidar  point  density  can  be  clearly  seen  using  point  density  maps, 
where  cells  are  color-coded  by  the  number  of  lidar  points  per  square  meter  (Figure  14). 
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Figure  13:  Lidar  Point  Patterns  and  Detection  of  Small  Features 


Simulated  craters,  Former  Camp  Beale  ESTCP  demonstration  site.  Overall  lidar  data  density  of  13.8  points  per 
square  meter.  All  points  surface  model  (left)  and  ground  point  surface  model  (right). 


Lidar  points  for  area  shown  above.  The  1 .5  m  test  Closer  view  of  image  to  left  showing  lidar  points,  1 .5  m 
crater  is  large  enough  to  intersect  several  lines  of  lidar  and  0.3  m  test  craters, 
points.  The  0.3  m  test  craters  are  small  enough  to  fit 
between  the  lines  of  lidar  points. 
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Figure  14:  Lidar  Point  Density  Variation 


Former  Camp  Beale  ESTCP  demonstration  site.  Point  density  maps  based  on  all  lidar  points  (center)  and 
ground  points  only  (right).  The  all  points  image  shows  flight  line  overlap  effects  (dark  horizontal  bands)  and 
aircraft  pitch  (narrow  dark  vertical  bands).  The  ground  points  image  shows  the  effects  of  point  classification 
method  (disappearance  of  flight  line  overlap  and  wind  effects)  and  of  vegetation  (lower  density  under  trees). 


4.1.4  Estimating  Confidence  Levels  for  Feature  Detection 

Regardless  of  the  overall  point  density  achieved,  lidar  point  density  will  always  vary 
somewhat  with  terrain  and  will  be  lower  under  vegetation.  This  will  affect  the  confidence 
levels  for  detection  of  ground  features  over  different  parts  of  the  site.  One  approach  to 
determining  confidence  levels  for  feature  detection  is  to  map  the  density  of  lidar  ground 
returns  (Figure  15).  Areas  with  no  returns  would  be  designated  as  areas  where  lidar 
would  not  detect  ground  features,  with  confidence  levels  increasing  with  the  number  of 
returns. 


Figure  15:  Preliminary  Confidence  Levels  for  Ground  Features 


Former  Camp  Beale  ESTCP  demonstration  site.  Point  density  map  based  on  reclassified  ground  points,  (vendor’s 
ground  points  plus  “low”  points  within  40  cm  of  the  original  ground  surface).  Scale  is  in  ground  points/m2. 
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4.2  Vegetation  Effects 

4.2.1  Lidar 

Laser  signals  do  not  penetrate  vegetation;  however,  lidar  often  can  be  used  successfully 
in  most  vegetated  conditions  (Figure  16).  This  is  because  the  laser  will  produce  a 
sufficient  number  of  signals  to  penetrate  the  many  small  gaps  in  the  foliage.  The  ability 
of  lidar  to  “look  through”  vegetation  is  enhanced  by  the  fact  that  modern  lidar  sensors 
can  receive  multiple  returns  from  the  same  laser  pulse. 


Figure  16:  Lidar  Surface  Models  under  Trees 


Former  Camp  Beale  ESTCP  demonstration  site.  Orthophoto  and  lidar-based  surface  model. 


Nevertheless,  higher-density  vegetation  will  always  result  in  fewer  laser  points  reaching 
the  ground  surface,  and  thus  in  a  less  dense  ground  model.  This  is  particularly  true  for 
low,  dense  brush  (Figure  17  and  18). 

4.2.2  Orthophotos 

Orthophotos  do  not  look  through  vegetation;  consequently  at  highly  vegetated  sites 
orthophotos  will  not  be  useful  for  detecting  ground  features.  At  such  sites,  it  may  be 
appropriate  to  use  pre-existing  orthophotos,  which  are  available  from  many  sources. 
While  pre-existing  orthophotos  often  have  larger  pixels  sizes  than  those  collected 
specifically  for  munitions  investigations,  they  are  still  appropriate  for  general  site  maps. 
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Figure  17:  Lidar  Surface  Models  in  Brush 


Former  Camp  Beale  demonstration  site,  orthophoto  and  lidar  in  mixed  vegetation.  The  lidar  image  is 
detailed  under  the  tree  cover  to  the  left,  but  degraded  in  the  thick  brush  to  the  lower  right. 


Figure  18:  Percentages  of  Vegetation  Returns 
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Percentage  of  lidar  points  reflected  from  vegetation  -  darker  green  and  blue  areas  have  higher  percentages  of  vegetation 
returns.  Although  much  ground  detail  is  visible  under  the  tree  cover,  the  lower  number  of  ground  returns  will  result  in  lower 
confidence  in  these  areas. 


4.3  Accuracy  and  Error  in  Lidar  Data 

Error  in  the  collection,  processing,  and  interpretation  of  lidar  data  can  lead  to  less 
accurate  site  characterization,  with  potential  for  both  false  negatives  and  false  positives 
in  the  detection  of  ground  features,  difficulty  in  integrating  lidar  with  other  spatial  data, 
and  incorrect  assessment  of  confidence  levels  in  the  data.  An  understanding  of  error  in 
lidar  data  can  contribute  to  appropriate  expectations  for  the  technology,  and  to  the 
development  of  appropriate  contract  specifications. 
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Understanding  lidar  error  is  especially  important  since  error  in  lidar  data  arises  from 
different  factors  than  error  in  magnetometry  or  EMI,  the  most  common  technologies 
used  in  geophysical  investigations  of  munitions  sites.  Magnetometry  and  EMI  directly 
detect  and  map  magnetic  anomalies  that  may  result  from  UXO,  and  each  anomaly  is 
reported  and  located  individually.  Error  is  understood  as  the  discrepancy  between  the 
reported  location  of  the  magnetic  anomaly  and  the  true  location  of  the  object. 

Lidar,  by  contrast,  uses  large  arrays  of  laser  reflections  to  model  surfaces.  The  location 
of  surface  features  (potential  craters,  for  instance)  is  then  inferred  from  characteristics  of 
the  entire  surface.  The  accuracy  of  the  location  of  features  in  the  modeled  surface  rests 
on  the  accuracy  and  precision  of  the  individual  lidar  points,  but  is  influenced  by  other 
factors  such  as  the  density  of  the  laser  returns,  the  characteristics  of  the  terrain,  and  the 
methods  used  to  create  the  surface  model.  These  factors  can  interact  in  complex  ways. 

4.3.1  Lidar  Point  Accuracy 

Lidar  vendors  guarantee  the  accuracy  of  lidar  points  as  a  part  of  their  contract 
documents,  and  vendors  quote  very  similar  guarantees  across  the  industry.  A  typical 
accuracy  specification  quoted  is  15  cm  vertical  and  50-100  cm  horizontal.5  Some 
vendors  provide  a  more  detailed  specification  such  as  the  following6.  (The  following 
values  are  at  95%  [two  sigma  or  two  standard  deviations]). 

■  Vertical:  15  cm  hard  surfaces  and  open  regular  terrain,  25  cm  soft/vegetated 
surfaces,  flat  to  rolling  terrain,  30-50  cm  soft/vegetated  surfaces,  hilly  terrain 

■  Horizontal:  50-75  cm  in  all  but  extremely  hilly  terrain  (depends  on  flying  height 
and  beam  divergence 

Lidar  vendors  guarantee  the  accuracy  of  the  lidar  data,  that  is,  its  correspondence  to 
surveyed  control.7  Accuracy,  in  contrast  to  precision,  is  the  closeness  of  an  estimated 
value  to  a  standard  or  accepted  correct  value.8  The  accuracy  value  refers  the  size  of  the 
differences  between  the  estimated  and  the  standard  value.  In  the  context  of  lidar,  the 
guaranteed  accuracy  refers  to  the  correspondence  of  surveyed  control  points  to  either 
the  individual  lidar  points  closest  to  the  surveyed  point,  or  to  the  elevation  of  the  lidar- 
derived  surface  model  at  that  point. 

Lidar  vendors  typically  express  their  stated  accuracies  as  root  mean  square  error 
(RMSE)  values.  In  calculating  RMSE,  the  difference  between  data  set  coordinate  values 
and  the  coordinate  values  from  an  independent  source  of  higher  accuracy  for  identical 
points  are  each  squared  and  then  averaged  over  the  sample,  after  which  the  square  root 
of  the  average  is  taken.  Since  the  errors  are  squared  before  they  are  averaged,  RMSE 
gives  a  relatively  high  weight  to  large  errors  compared  to,  for  instance,  an  accuracy 


5  See:  http://www.airborne1  .com/technology/LiDARAccuracy.pdf 

6 ,  Available  from  Terrapoint,  LLC.  See:  http://www.ambercore.com/files/TerrapointWhitePaper.pdf 

7  Surveyed  control  points  are  often  assumed  to  be  free  of  error,  which  in  really  will  not  be  true  since  control 
points  are  surveyed  using  techniques  and  instruments  that  are  themselves  subject  to  error.  Control  points, 
therefore,  should  be  reported  with  an  appropriate  error  measure,  which  can  be  considered  in  evaluating  the 
accuracy  of  the  lidar  points. 

8  The  discussion  in  this  section  is  adapted  from  Chapter  3  of  Digital  Elevation  Model  Technologies  and 
Applications:  The  DEM  Users  Manual  (ASPRS  2001). 
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measure  such  as  the  mean  absolute  error,  which  gives  the  same  weight  to  all  values. 
This  means  that  RMSE  is  most  useful  when  large  errors  are  particularly  undesirable. 

The  error  of  some  lidar  points  will  always  fall  outside  of  the  stated  accuracy  range. 
Consequently,  RMSE  values  are  generally  quoted  at  the  68%  (one  sigma  or  one 
standard  deviation)  or  95%  (two  sigma  or  two  standard  deviations)  level.9 

One  weakness  of  RMSE  is  that,  as  a  single  number,  it  does  not  capture  the  spatial 
variability  of  lidar  point  error.  As  discussed  in  Section  4.3.4,  lidar  error  can  vary  with 
terrain  and  other  factors,  and  RMSE  alone  will  not  capture  this  variation.  Useful 
supplements  to  RMSE  would  include  maps  of  principal  terrain  and  vegetation  types,  and 
calculation  of  error  for  each  principal  terrain  type  separately.  This  is  the  approach  used 
by  the  Federal  Emergency  Management  Agency  (FEMA)  in  its  accuracy  specification  for 
lidar  collected  in  support  of  FEMA’s  flood  mapping  program  (FEMA  2003). 

4.3.2  Assessing  the  Lidar  Point  Accuracy 

Horizontal  and  vertical  accuracy  of  lidar  points  are  assessed  differently.  To  measure 
vertical  accuracy,  lidar  vendors  commonly  survey  a  variety  of  locations  in  the  study  area, 
and  compare  these  surveyed  elevations  to  lidar  elevations  at  the  same  points.  Surveyed 
points  are  commonly  established  using  static  survey  methods.  Some  vendors  will 
supplement  static  surveyed  points  with  large  numbers  of  additional  points  collected 
using  kinematic  GPS  survey  methods,  collected  by  driving  along  roads  in  the  project 
area.  Other  vendors  use  small  unmanned  rover  vehicles  to  collect  large  numbers  of 
static  points. 

To  evaluate  horizontal  accuracy,  vendors  compare  the  location  of  the  lidar  returns  for 
objects  in  the  study  area  with  real-world  horizontal  locations  that  can  be  surveyed. 

These  may  include  target  objects  placed  with  the  surveyed  points  as  in  Figure  19,  or 
larger  objects.  For  example,  lidar  elevation  values  can  be  used  to  model  the  corners  of 
buildings  and  other  structures,  or  lidar  intensity  values  can  be  used  to  model  the  edges 
of  pavement  (Figure  20).  These  lidar-based  locations  can  then  be  compared  to 
surveyed  locations  for  the  same  objects.  These  same  objects  can  be  used  to  assess 
lidar  to  orthophoto  alignment. 


Figure  19:  Surveyed  Points  Used  to  Assess  Lidar  Vertical  Error 


9  Formulas  for  calculating  RMSE  can  be  found  in  ASPRS  (2001). 
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Typical  surveyed  points  used  to  evaluate  vertical  and  horizontal  accuracy. 


Figure  20:  Objects  Used  to  Asses  Lidar  Horizontal  Error 


Lidar  points  reflecting  on  and  around  a  pre-placed  rectangular  control  surface.  Lidar  points  are  color-coded  to 
show  reflections  from  the  ground  surface  (green)  and  those  at  or  above  the  known  height  of  the  flat  panel  (red). 


Lidar  elevation  values  are  used  to  model  the  edge  of  the 
building.  The  modeled  location  is  compared  to  the 
surveyed  location  -  not  to  the  location  in  the  orthophoto 
since  this  may  be  subject  to  different  sources  of  error. 


Lidar  intensity  values  are  used  to  model  the  edges 
of  pavement  (as  in  the  runway  above).  Locations  of 
pavement  edges  or  corners  are  compared  surveyed 
locations. 


Surveyed  control  points  provide  survey-grade  data  points  of  a  higher  accuracy  than  the 
lidar  data  to  assess  the  accuracy  of  the  lidar  points.  Vendors  may  perform  vertical 
adjustments  of  the  entire  lidar  data  set  to  achieve  a  best  possible  fit  to  the  control 
points10,  in  which  case  the  residual  differences  after  this  adjustment  provide  the 


10  At  least  one  vendor  state  that  without  such  adjustment,  vertical  errors  of  the  lidar  data  would  routinely 
exceed  the  manufacturers  reported  maximum  vertical  error  of  15  cm  (Sky  Research,  2009). 
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quantitative  estimate  of  the  vertical  error  of  the  lidar  data.  The  surveyed  points  are  also 
used  to  validate  the  accuracy  of  the  lidar  system  and  to  validate  the  lidar  calibration 
values. 

Both  surveyed  control  points  and  lidar  point  locations  depend  on  GPS  to  determine 
locations.  It  is  therefore  important  not  to  have  too  great  a  distance  between  the  study 
area  and  the  nearest  GPS  base  station  that  will  be  used  for  post-processing.  Generally, 
50  kilometers  is  considered  a  reasonable  limit,  after  which  the  accuracy  of  the  lidar  data 
begins  to  degrade.  Most  airports  have  GPS  base  stations;  however,  a  closer  base 
station  should  be  established  for  sites  located  at  a  sufficient  distance  from  the  airport.  In 
remote  areas  where  established  GPS  base  stations  are  scarce  or  non-existent,  lidar 
acquisition  may  require  support  from  helicopters  to  establish  control  points  and 
additional  base  stations  in  or  near  the  project  area. 

Horizontal  and  vertical  accuracy  are  reported  by  vendors  using  standard  quality  control 
reports.  Table  1  shows  an  example  lidar  vertical  error  report  from  a  vendor,  displaying 
lidar-to-control-point  vertical  error  for  eight  control  points.  It  shows  an  error  of  just  over 
10  cm  or  0.104  meter  (m)  at  the  95%  confidence  level. 


Table  1 :  Example  Lidar  Vertical  Error  Report  from  a  Vendor 


Statistics 

Target 

Easting 

Northing 

Survey 

Elevation 

Lidar  Elevation 

Difference 

TAR1 

543461 .243 

3805793.824 

827.486 

827.440 

-0.046 

TAR2 

542947.381 

3807961.251 

837.761 

837.740 

-0.021 

TAR3 

543747.347 

3809712.492 

833.970 

833.930 

-0.040 

TAR4 

545277.331 

3809859.694 

797.753 

797.710 

-0.043 

TAR5 

546671 .936 

3810138.520 

846.280 

846.230 

-0.050 

TAR6 

547168.309 

3808650.964 

864.049 

864.000 

-0.049 

TAR7 

545682.049 

3807795.187 

784.720 

784.650 

-0.070 

TAR8 

546462.139 

3805793.466 

806.263 

806.180 

-0.083 

Summary 

Average  difference 

-0.050 

Minimum  difference 

-0.083 

Maximum  difference 

-0.021 

RMSE  of  the  elevation 

+/-  0.053 

2d  RMSE  of  the  elevation  (95%) 

+/- 0.104 

Std-Dev.  of  the  elevation 

+/- 0.019 

Source:  Terra  Remote  Sensing 

Surveyed  and  lidar  elevations  in  this  chart  are  reported  in  meters  above  the  “height  above  the  geoid”,  an  elevation  that 
roughly  coincides  with  mean  sea  level.  See  ASPRS  (2001 )  for  additional  discussion  of  elevation  values  in  GPS  and 
lidar. 

4.3.3  Lidar  Point  Precision 

Often  confused  with  accuracy,  precision  is  a  measure  of  the  tendency  of  a  set  of  values 
to  cluster  about  a  number  determined  by  the  set.  The  usual  measure  of  precision  is  the 
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standard  deviation  or  the  standard  error.  Precision  is  distinguished  from  accuracy  in  that 
accuracy  is  a  measure  of  the  tendency  to  cluster  about  a  value,  such  as  a  surveyed 
point,  which  is  not  determined  by  the  data  set  but  specified  in  some  other  manner  (such 
as  ground  survey).  Therefore,  in  order  to  be  highly  precise,  a  data  set  needs  only  to 
conform  to  itself,  while  to  be  accurate  the  data  set  must  conform  to  an  independently 
derived  standard  (ASPRS  2001). 

If  lidar  precision  were  perfect,  there  would  be  no  elevation  differences,  for  instance, 
between  lidar  returns  on  a  uniformly  flat  surface.  In  practice,  precision  is  never 
completely  perfect,  and  precision  errors  appear,  for  instance,  as  the  “texture”  observable 
on  paved  parking  areas  or  roads  (Figure  21 ).  Precision  error  of  this  type  may  arise  from 
small  errors  in  the  sensor  system,  and  to  this  extent  reflects  the  inherent  limits  of  the 
technology  at  its  current  state  of  development,  or  precision  errors  may  arise  from 
insufficiently  tight  calibration,  which  can  lead  to  higher  than  necessary  vertical  distances 
between  lidar  points. 


Figure  21:  Lidar  Precision  Error 


Orthophoto  and  lidar  surface,  municipal  airport,  Washington  State.  The  lidar  surface  shows  more  “texture”  than 
the  orthophoto  on  the  flat  surfaces  of  the  parking  lot  and  building  roofs. 


4.3.4  Sources  of  Lidar  Point  Error 

Several  factors  influence  the  accuracy  and  precision  of  individual  lidar  points.  In  most 
cases,  the  magnitude  of  each  error  source  is  not  well  documented,  since  vendors  and 
manufacturers  generally  report  error  values  for  the  system  as  a  whole. 

Instrument  error.  Broadly,  instrument  error  refers  to  the  difference  between  the  value 
given  by  an  instrument  and  the  “actual”  values,  based  on  the  accuracy  and  precision  of 
the  measuring  instrument.  In  the  context  of  lidar,  instrument  error  consists  of  deviations 
from  positional  values,  generally  as  determined  by  GPS-based  survey  methods, 
contributed  by  each  physical  component  of  the  system.  Instrument  error  can  impact 
both  accuracy  and  precision. 

Manufacturers  and  vendors  typically  report  error  for  the  entire  system,  expressed  as  the 
difference  between  the  positions  of  individual  lidar  points  compared  to  surveyed 
positions.  Reported  error  for  one  commonly-used  lidar  system,  the  Leica  Geosystems 
ALS60  Airborne  Laser  Scanner,  is  presented  below.  This  system  is  comparable  to 
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others  in  common  use  and  represents  typical  error  values.  The  error  curves  are  based 
on  a  40-degree  field  of  view  and  a  nominal  5  cm  GPS  error. 

Figure  22  shows  that  vertical  (z)  error  ranges  from  under  10  cm  to  over  20  cm 
depending  on  flight  height  and  distance  from  nadir,  and  that  horizontal  (x,y)  error  ranges 
from  around  5  cm  to  over  60  cm,  again  depending  on  flight  height  and  distance  off  nadir. 
Most  lidar  collected  at  munitions  sites  has  been  collected  at  flight  heights  of  300-1 ,000 
m,  where  both  horizontal  and  vertical  error  is  lower. 

Figure  22:  Leica  ALS60  Horizontal  and  Vertical  Error  Curve 


0  1000  2000  3000  4000  5000  6000 

Flight  Height  (m) 

Site  Conditions.  A  wide  variety  of  local  conditions  can  create  error  in  lidar  data.  Some 
conditions  affect  the  entire  data  set,  either  by  affecting  the  equipment  directly  or 
interfering  with  the  GPS  signal.  Others,  such  as  slope,  may  affect  only  part  of  the  survey 
area.  Site  conditions  that  can  contribute  to  lidar  error  include: 

■  Slope.  Lidar  collected  on  steep  terrain  will  be  less  accurate  than  that  on  flat 
terrain  (Figure  23).  This  error  arises  in  two  ways: 

-  The  lidar  pulse  spreads  as  the  beam  travels.  The  degree  of  beam 

divergence  is  controlled  by  the  characteristics  of  the  laser  combined  with  the 
flight  height,  with  typical  beam  footprints  between  10  and  100  cm  in 
diameter.11  Beam  divergence  creates  an  area  of  uncertainty  as  to  the  exact 
point  where  enough  energy  is  reflected  to  trigger  a  return.  On  flat  surfaces 
this  uncertainty  may  affect  the  horizontal  location  of  the  return,  but  will  not 
affect  the  elevation  value.  On  sloping  surfaces,  the  area  of  the  pulse  footprint 
will  be  larger,  and  will  include  a  vertical  error  range. 


1 1 

The  formula  for  beam  divergence  is  roughly:  spot  diameter  at  nadir  =  elevation  (meters)*beam 
divergence  (radians)  (Baltsavias  1999).  System  manufactures  report  beam  divergence  factors  from  0.22  to 
1 .0  mrad  (Key  2009).  Laser  footprints  would  therefore  range  from  22  to  100  cm  at  1 ,000  m  flight  elevations. 
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-  The  horizontal  error  of  the  lidar  location,  caused  by  both  beam  divergence 
and  the  error  inherent  in  the  GPS,  IMU,  and  other  components  will  be  larger 
on  sloped  ground.  Further,  on  sloped  ground  the  horizontal  error  will  magnify 
the  vertical  error  versus  flat  ground.  The  maximum  amount  of  elevation  error 
introduced  is  a  function  of  surface  slope,  with:  Elevation  Error  =  tan  a  x 
Horizontal  Displacement. 


Figure  23:  Effects  of  Slope  on  Lidar  Point  Error 


■  Type  of  ground  surface.  Some  vendors  guarantee  higher  accuracy  on  hard 

surfaces  than  on  soft  ground  surfaces;  this  is  based  on  the  fact  that  soft  surfaces 
are  often  less  clearly-definable.  Plowed  fields  or  grassy  areas,  for  instance,  have 
furrows  or  vegetation  that  would  create  error  up  to  10-20  cm  compared  to  use  of 
a  survey  pole  (Neufeldt  2009). 

Similarly,  highly  reflective  surfaces  occasionally  appear  to  be  slightly  raised  in 
comparison  to  non-reflective  surfaces,  such  as  white  painted  centerlines  on 
asphalt  roadways.  It  is  possible  that  more  reflective  surfaces  are  returning 
sufficient  energy  to  trigger  a  response  more  quickly  than  less  reflective  surfaces. 
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■  Vegetation  conditions.  Under  vegetated  conditions,  it  is  common  for  the 
majority  of  laser  signals  to  reflect  from  the  vegetation  rather  than  the  ground 
surface  (Figure  18).  The  lower  density  of  lidar  points  under  vegetation  will  lead  to 
lower  confidence  levels  for  detection  of  small  features.  The  accuracy  of  the 
individual  lidar  points  under  vegetation  should  not  be  affected;  however,  the 
ground  surface  model  will  be  coarser,  which  may  lead  to  more  discrepancy 
between  the  modeled  surface  and  surveyed  points  at  any  particular  location. 

■  Electromagnetic  interference.  Rarely,  certain  types  of  man-made 
electromagnetic  signals  can  affect  lidar  data  collection.  Such  interference  was 
noted  at  the  Former  Camp  Beale  ESTCP  demonstration  site.  Beale  Air  Force 
Base,  which  is  adjacent  to  the  demonstration  site,  is  the  site  of  one  of  three 
installations  that  are  part  of  the  Phased  Array  Warning  System  radar  system, 
designed  to  detect  and  track  sea-launched  ballistic  missiles.  The  high-intensity 
radar  signals  from  this  installation  disrupted  the  GPS  time  signal  used  by  the  lidar 
and  orthophoto  sensors,  initially  rendering  the  data  unusable.  The  effect  was 
noted  within  several  kilometers  of  the  radar  station  and  at  altitudes  up  to 
approximately  500  m.  The  problem  was  noted  during  the  daily  QA/QC  checks  on 
the  first  day  of  data  collection.  A  sample  of  the  data  was  sent  to  the  vendor’s 
office,  and  a  solution  was  developed  to  re-insert  the  correct  GPS  times. 
Interestingly,  lidar  was  subsequently  collected  at  Beale  Air  Force  Base  without 
problems,  since  lidar  data  collection  took  place  at  a  higher  altitude  outside  of  the 
influence  of  the  radar  system. 

■  Geomagnetic  activity.  A  different  form  of  electromagnetic  interference, 
geomagnetic  activity  refers  to  natural  variation  in  the  earth’s  magnetic  field, 
usually  as  a  result  of  solar  flares  and  other  solar  activity.  During  periods  of  high 
geomagnetic  activity  the  GPS  phase  can  be  affected  to  the  point  where  the 
receiver  cannot  perform  phase  measurements  with  enough  precision  for 
centimeter  level  accuracy.  It  is  good  practice  for  vendors  to  check  Government 
web  sites  for  geomagnetic  activities  prior  to  data  collection. 

■  Weather  and  temperature.  Weather  and  temperature  conditions  can  affect  the 
ability  to  collect  lidar  data,  but  should  not  affect  the  accuracy  or  precision  of  the 
lidar  data  itself. 

4.3.5  Creating  Surface  Models 

In  practice,  much  of  the  analysis  of  lidar  data  is  performed  using  surface  models  derived 
from  the  lidar  points.  Several  factors  influence  the  usability  of  lidar-based  surface 
models;  some  are  due  to  error  and  others  to  technical  choices  that  may  be  inappropriate 
to  the  site. 

The  typical  final  products  of  lidar  are  digital  models  of  the  ground  surface,  both  of  the 
bare  ground  and  the  ground  with  vegetation  and  buildings  included.  Typical  model  types 
are  DEMs,  digital  surface  models  (DSMs),  and  DTMs. 

A  DEM,  as  defined  by  the  USGS,  is  a  digital  file  consisting  of  terrain  elevations  for 
ground  positions  at  regularly-spaced  horizontal  intervals  that  portrays  the  ground  surface 
free  of  vegetation  or  human-created  structures.12  DEMs  may  be  created  using  data  from 


12  See:  http://rockyweb.cr.usgs.gov/elevation/dpi_dem.html 
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many  sources  in  addition  to  lidar,  including  topographic  maps,  ground  survey, 
photogrammetry,  or  synthetic  aperture  radar.  In  the  context  of  lidar,  the  DEM  is  the 
product  through  which  the  semi-random  mass  lidar  points  that  the  vendor  classifies  as 
returning  from  the  ground  surface  are  converted  to  a  regularly-spaced  grid  of  elevation 
values.  This  digital  model  can  then  be  used  in  standard  GIS  or  other  software  to 
produce  hillshaded  surfaces,  contour  lines,  or  other  digital  products. 

Regularly  spaced  elevation  files  of  this  type  also  may  be  created  using  all  returns, 
including  those  from  the  tops  of  buildings,  trees  and  other  features.  The  USGS  refers  to 
these  as  DSMs.  The  term  DTM  is  used  as  a  synonym  for  both  DEM  and  DSM.  In  the 
lidar  context,  the  term  DTM  is  most  frequently  used  to  refer  to  the  all-points  surface 
model.  This  paper  uses  the  term  DSM  for  the  all-points  surface  model  including  both 
ground  and  non-ground  returns. 

Error  in  the  DEM  can  be  defined  as  the  discrepancy  between  the  elevation  values  of  the 
DEM  cells  compared  to  surveyed  values  at  the  same  locations.  Error  in  the  elevation 
values  of  the  DEM  cells  also  affects  the  horizontal  error  of  features  visible  in  the  ground 
surface  model,  since  it  is  the  elevation  differences  between  adjacent  DEM  cells  that 
create  edges  that  allow  the  user  to  infer  the  location  of  features. 

Factors  Affecting  Surface  Models.  The  accuracy  and  usefulness  of  a  surface  model 
will  depend  on  the  accuracy  of  the  lidar  data  from  which  it  is  created,  and  will  be  subject 
to  the  types  of  error  that  affect  lidar  points.  However,  the  usefulness  of  the  model  can 
be  impacted  by  the  methods  used  to  create  the  model  itself,  including: 

■  The  choice  of  the  points  classified  as  ground  or  non-ground  returns 

■  The  choice  of  interpolation  method  from  the  lidar  points  to  the  individual  cell 
elevations 

■  The  choice  of  cell  size 

Point  classification  methods.  The  point  classification  approach  used  by  the  lidar 
vendor  should  have  no  effect  on  the  accuracy  or  precision  of  the  lidar  points,  since 
classification  takes  place  after  the  data  is  calibrated.  However,  methods  that  are  biased 
towards  creating  clean,  smooth  ground  surfaces  can  result  in  a  lower  data  density  of 
ground  points,  and  this  can  result  in  fewer  detections  of  small  ground  features. 

Interpolation  method.  The  technical  problem  in  creating  surface  models  from  lidar 
points  is  to  assign  an  accurate  elevation  value  to  each  regular  grid  cell  from  the  semi¬ 
random  lidar  points.  There  are  two  general  approaches  to  assigning  these  values.  The 
first  approach  is  to  interpolate  the  cell  values  directly  from  the  lidar  points.  There  are  a 
variety  of  mathematical  approaches  to  such  interpolation  available  in  GIS  programs  and 
other  software.13 

The  second  approach  is  to  create  a  triangulated  irregular  network  (TIN)  using  every 
individual  lidar  point.  The  TIN  is  then  used  to  create  the  surface  model,  again  using  one 
of  the  interpolation  methods  described  above.  Either  approach  can  be  employed  using 
the  ground  points  only  or  both  the  ground  and  vegetation  points. 


13  These  methods  are  described  in  detail  in  references  such  as  the  ArcGIS  help  files. 
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Both  of  the  above  approaches  produce  useable  surfaces,  with  each  method  having  its 
own  particular  advantages  and  shortcomings.  However,  the  TIN  method  appears  to  be 
the  most  common  approach  at  the  time  of  this  document  (Figure  24). 

Cell  size.  Choice  of  cell  size  can  affect  the  detection  of  small  surface  features 
dramatically  (Figure  25),  with  smaller  cell  sizes  performing  better.  Vendors  and 
researchers  suggest  that  cell  size  be  no  larger  than  the  average  ground  point  spacing, 
and  that  use  of  cells  smaller  than  the  average  point  spacing  may  not  sacrifice  accuracy. 

Display  methods.  Surface  models  are  usually  displayed  as  hillshaded  images  to 
enhance  the  visualization  of  surface  features  (Figure  26).  ArcGIS,  the  most  commonly 
used  GIS  software,  contains  dozens  of  settings  that  can  affect  the  usefulness  of  these 
images.  Different  settings  may  be  appropriate  in  some  circumstances,  and  users  need 
to  work  with  analysts  both  to  fully  understand  the  goals  of  the  survey  and  to  experiment 
with  appropriate  settings. 


Figure  24:  Process  Steps  for  TIN-Based  Lidar  Surface  Models 
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Figure  25:  Effects  of  Surface  Model  Cell  Size  on  Feature  Detection 
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Figure  26:  Hillshade  Display  Method  Examples 


Former  Camp  Beale  ESTCP  demonstration  site,  Hillshade  image,  alternate  settings  showing  less 
hilshade  image,  ArcGIS  default  settings.  The  default  dark  grey  area  but  also  less  contrast  on  the  craters 
settings  are  too  dark  in  some  areas  to  reveal  the 
ground  surface. 
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Hillshade  image,  “neutral”  hillshade  settings 


Hillshade  image,  contrast  boosted,  highlighting 
craters  but  obscuring  other  detail 


4.4  Accuracy  and  Error  in  Orthophotos 

Positional  error  in  orthophotos  can  be  expressed  as  the  difference  between  the  apparent 
location  of  the  orthophoto  pixel  and  its  true  ground  location.  Positional  error  is  specified 
in  contract  documents  and  verified  by  comparing  objects  in  the  image  to  surveyed 
locations,  as  discussed  in  Section  4.3.2. 

In  addition  to  positional  error,  there  are  a  number  of  other  quality  issues  that  can  arise  in 
the  creation  and  use  of  orthophotos,  including: 

■  Color  balance.  Vendors  will  sometimes  have  a  difficult  time  achieving  a  good 
color  balance  among  the  numerous  digital  images  in  the  mosaic.  While  it  is 
difficult  to  establish  a  rigorous  contract  specification,  color  balance  should  be 
inspected  on  receipt  of  the  orthophotos  and  any  problems  discussed. 

■  Position  of  base  vs.  top  of  objects.  In  parts  of  the  image  that  are  not  directly 
under  the  aircraft,  the  image  will  include  portions  of  the  sides  of  tall  objects  such 
as  trees  and  buildings.  In  such  cases,  the  top  of  the  image  is  not  located  directly 
over  the  base  of  the  object,  and  while  the  base  of  the  object  will  be  correctly 
spatially  located,  the  top  will  appear  slightly  offset  (Figure  27).  This  is  a  limitation 
of  the  technology  which  should  be  understood  by  users. 
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Figure  27:  Apparent  Misalignment  of  Tops  of  Buildings  in  Orthophotos 


Airport  site,  Washington  State.  Orthophoto  and  lidar  compared  to  surveyed  building  locations.  The  orthophoto  on  the 
left  shows  slight  apparent  misalignment  of  the  top  of  the  buildings  compared  to  surveyed  locations,  while  the  lidar 
image  on  the  right  (color  coded  by  elevation)  shows  better  alignment. 


■  “Leaning”  trees.  Another  implication  of  portions  of  the  image  being  off-nadir  is 
that  groups  of  objects,  most  often  trees,  can  appear  to  lean,  sometimes  in 
paradoxical  ways.  This  problem  is  greater  with  larger  image  sizes,  where  the  off- 
nadir  angle  will  be  greater,  and  also  in  steep  terrain.  The  problem  can  be 
minimized  by  skillful  assembly  of  the  image  mosaic,  but  cannot  be  eliminated 
entirely. 

■  Shadows.  Shadows  in  orthophotos  can  sometimes  be  extreme  and  can  obscure 
ground  features.  Shadows  can  be  minimized  by  collecting  imagery  at  times  of 
day  and  times  of  the  year  when  the  sun  is  close  to  overhead,  and  vendors  will 
normally  plan  data  acquisition  for  such  times  when  possible.  For  relatively  flat 
munitions  sites  without  tall  vegetation,  however,  images  collected  at  low  sun 
angles  can  have  some  advantages  since  the  resulting  shadows  will  highlight 
small  craters,  dirt  roads,  and  other  faint  objects  (Figure  28). 
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Figure  28:  Shadows  in  Orthophotos 


Nevada  desert  site:  steep  canyons  create  deep  shadows  in  the  orthophotos;  these  areas  are  largely  unusable. 
Images  could  have  been  improved  by  flying  during  higher  sun  angle. 


Navy  Range  Boardman,  OR:  orthophotos  taken  at  low  sun  angle,  shadows  enhance  the  visibility  of  ground  features. 
Left,  vehicle  tracks  can  be  seen  to  the  south  of  the  dirt  road.  Right,  the  road  is  much  more  visible  as  a  result  of  the 
shadowing  at  its  edges,  which  would  not  have  been  present  if  the  sun  had  been  at  or  near  nadir. 


4.5  Operator  Error 

Potential  areas  where  actions  of  the  operator  can  affect  lidar  products  include  mission 
planning,  field  data  collection  practices,  data  calibration,  and  lidar  point  classification. 
There  are  many  sources  of  operator  error,  including  inexperience  or  improper  training, 
momentary  lack  of  judgment,  inattentiveness,  or  distraction.  Operator  error  also  may 
result  from  incomplete  information,  as  when  the  operator  does  not  fully  understand  the 
purpose  of  the  lidar  survey.  Operator  error  is  difficult  or  impossible  to  quantify. 
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The  standard  approach  to  eliminating  operator  error  is  through  the  application  of  training 
programs  to  reduce  error  and  quality  control  programs  to  detect  and  correct  errors.  Most 
vendors  have  well-developed  training  and  QA  programs,  and  the  impact  of  serious 
operator  error  is  generally  small.  This  is  particularly  true  in  regard  to  all  processes  that 
impact  the  positional  accuracy  of  the  lidar,  where  operator  error  may  cause  the  vendor  to 
fail  to  meet  contract  specifications. 

4.6  GPS  Error 

GPS  error  is  a  component  of  the  overall  lidar  system  error  discussed  above.  Error  in 
GPS  is  discussed  separately  first,  since  vendors  report  that  GPS  error  is  the  largest 
source  of  error  in  the  day-to-day  use  of  lidar  technology,  and  second,  since  GPS  error 
affects  not  only  the  positional  accuracy  of  the  lidar  points,  but  the  accuracy  of  field  data, 
including  both  calibration  points  and  the  reported  locations  of  features  in  the  field. 

There  are  three  broad  categories  of  GPS  receivers  on  the  market.  Each  has  a  different 
level  of  positional  accuracy,  and  this  should  be  accounted  for  in  reporting  the  positions 
established  in  the  field.  GPS  accuracy  is  degraded  under  vegetation  for  all  types  of 
equipment.  Receiver  types  include: 

■  Recreation  or  consumer-grade  GPS,  which  have  quoted  positional  accuracies  of 
3-5  m  in  open  areas.14  Manufacturers  do  not  quote  accuracies  for  obstructed 
areas  such  as  under  vegetation  canopy  for  any  type  of  GPS  receiver  (Clarkin 
2007);  however,  Wing  and  Eklund  (2007)  found  recreation-grade  receivers  to  be 
capable  of  accuracies  within  10  m  under  closed  canopies  and  7  m  under  young 
forest  in  western  Oregon. 

■  Resource  or  mapping-grade  receivers,  which  have  a  quoted  accuracy  in  the  open 
as  sub-meter  down  to  around  30  cm.15  Researchers  have  found  these  single¬ 
frequency  receivers  to  achieve  error  between  0.2  and  6  m  under  vegetation, 
depending  on  conditions  (Clarkin  2007). 

■  Survey  grade  receivers,  which  achieve  sub-centimeter  accuracy  in  open 
conditions.  Hasegawa  and  Yoshimura  (2003)  found  that  survey  grade  receivers 
achieve  accuracies  of  0.02-0.4  m  under  dense  canopy.  Naesset  (2001)  found  a 
survey-grade  dual  frequency  receiver  had  mean  positional  accuracies  of  0.08- 

1 .35  m  under  dense  canopy. 

4.7  Summary:  Implications  for  Use  of  Lidar  and  Orthophotos 

The  preceding  sections  have  several  implications  for  the  use  of  lidar  and  orthophotos  at 
munitions  sites. 


14  For  example,  see  http://www.magellangps.com/products/product.asp?seglD=425&prodlD=1916  for  the 
Magellan  Triton  2000  series,  a  high-end  hand-held  model. 

15  See  http://www.trimble.com/pathfinderproxt.shtml  or 

http://store.elecdata.com/trimble/pathfinder_proxh_receiver.aspx.  For  example,  the  Trimble  R8  GNSS,  a 
commonly-used  GPS  system  in  both  static  GPS  and  real  time  kinematic  surveying,  has  published  accuracy 
specifications  of  ±5  mm  +0.5  ppm  horizontally  and  ±5  mm  +1  ppm  vertically  for  static  GPS  applications,  and 
±1  cm  +1  ppm  horizontally  and  ±2  cm  +1  ppm  vertically  for  real  time  kinematic  applications.  See  Trimble  R8 
GNSS  System  Datasheet,  http://trl.trimble.com/docushare/dsweb/Get/Document-140079/022543- 
079H_TrimbleR8GNSS_DS_0309_LR.pdf. 
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Implications  for  calibration  and  quality  control.  Errors  in  the  surveyed  calibration 
points  can  lead  to  incorrect  assessment  of  data  quality  or  errors  in  the  entire  lidar  data 
set.  Surveyed  points  used  for  calibration  and  quality  control  must  be  from  a  source  of 
higher  quality  than  the  lidar  data  itself.  Standard  practice  in  establishing  calibration 
points  includes: 

■  Using  survey-grade  GPS  operated  by  qualified  staff. 

■  Choosing  flat  sites  to  minimize  the  effects  of  slope  between  surrounding  control 
point  and  nearest  lidar  points. 

■  Choosing  open-sky  sites,  which  are  essential  to  GPS  quality. 

■  Choosing  calibration  points  in  all  major  terrain  types.  Actual  data  calibration  will 
be  performed  using  the  highest  quality  surveyed  points,  which  may  exclude 
points  in  especially  rugged  or  soft  terrain.  However,  points  in  all  terrain  types  will 
be  used  for  quality  control  and  accuracy  evaluation. 

GPS  error  in  field  surveys.  In  most  field  work,  resource-grade  GPS,  with  its  sub-meter 
accuracy,  should  be  able  to  locate  munitions-related  ground  features  with  sufficient 
accuracy  to  compare  the  with  lidar  data,  as  long  as  open-sky  conditions  are  present.  At 
vegetated  sites  where  GPS  error  can  be  larger,  more  complex  methods  to  locate 
positions  may  be  needed. 

Surface  models.  In  the  creation  of  digital  surface  models,  vendors  and  Government 
end  users  have  several  options  to  assure  that  surface  models  best  serve  the  needs  of 
the  investigation: 

■  Vendors  should  be  required  to  meet  tight  calibration  standards  to  minimize 
elevation  differences  that  may  result  in  over-classification  of  non-ground  points. 
Optimum  point  classification  methods  for  munitions  sites  have  not  yet  been 
developed;  however,  the  appropriate  principle  is  to  retain  the  largest  number  of 
points  possible  in  the  ground  classification,  even  at  the  cost  of  including  some 
additional  surface  noise. 

■  In  creating  DEMs  and  DTMs,  vendors  and  Government  end  users  should  use  the 
smallest  cell  size  consistent  with  data  density. 

■  Vendors  and  end  users  should  consider  experimenting  with  interpolation 
methods. 

In  practice,  creating  DEMs  and  DTMs  is  more  conveniently  accomplished  by 
Government  end-users  in-house,  rather  than  having  the  vendor  deliver  them,  since 
creation  in-house  can  allow  for  experimentation  with  alternative  methods.  However,  this 
requires  that  Government  end-users  have  sufficient  software  tools  and  training  to 
accomplish  these  tasks.  Creating  DEMs  and  DTMs  in-house  can  be  challenging, 
especially  with  high-density  data  sets  where  the  number  of  points  is  very  large.  In  such 
cases  it  may  be  appropriate  to  work  with  sample  data  sets  to  determine,  in  consultation 
with  vendors,  the  appropriate  specifications  for  these  products,  and  then  request  that  the 
vendor  create  and  deliver  the  final  products. 
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5  Costs  and  Contracting  Considerations 
5.1  Factors  Affecting  Cost 

Costs  for  lidar  and  orthophoto  surveys  are  variable  and  depend  on  site-specific  factors. 
However,  a  planning-level  cost  of  between  $10  and  $30  per  acre  is  typical  at  the  time  of 
this  report  (URS  2009b).  This  cost  includes  acquiring  lidar  in  the  range  of  four  to  six 
points  per  square  meter,  orthophotos  at  a  10-20  cm  pixel  size,  creating  standard  GIS 
products,  and  performing  QA/QC  review  and  initial  analysis.  Table  2  presents  the 
principal  factors  affecting  estimated  cost,  and  whether  these  factors  also  affect 
performance. 


Table  2:  Factors  Affecting  Cost  and  Performance 


Parameter 

Value 

Cost  Impact 

Performance  impact 

Data  density 

Orthophotos  not 
collected 

Cost  approximately  25% 
lower  than  the  range 
quoted 

Project  must  rely  on  pre¬ 
existing  orthophotos 

Orthophotos  ~  20  cm 
pixels  or  larger 

Cost  on  the  lower  end  of 
the  range  quoted 

Orthophtos  appropriate  for 
overall  site  maps  but  limited 
usefulness  for  individual 
object  detection 

Orthophotos  -10  cm 
pixels 

Costs  on  the  higher  end  of 
the  range  quoted 

Orthophotos  appropriate  for 
both  overall  site  maps  and 
individual  object  detection 

Lidar  ~  4-6  pts/m2 

Costs  on  the  lower  end  of 
the  range  quoted 

Surface  models  appropriate 
for  detection  of  target 
objects  and  larger  individual 
objects 

Lidar  -8-10  pts/m2 

Costs  on  the  higher  end  of 
the  range  quoted 

Surface  models  appropriate 
for  detection  of  target 
objects  and  larger  numbers 
of  individual  objects 

Weather 

Clear 

Cost  not  affected 

Data  quality  not  affected 

Wind,  rain,  or  snow 

Additional  labor  and  per 
diem  for  each  day  that 
weather  prevents  data 
collection 

Data  cannot  be  collected  in 
wind,  rain,  or  snow 

Summer  temperatures 
over  100°  F 

Cost  of  additional  labor, 
per  diem  and  aircraft  rental 
due  to  shorter  data 
collection  flights 

Data  quality  not  affected, 
but  data  collection  limited  to 
early  morning  hours 
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Parameter 

Value 

Cost  Impact 

Performance  impact 

Vegetation 

No  vegetation  or  low 
grass 

No  impact 

Lidar:  no  impact 
Orthophotos:  no  impact 

High  grass  and  light 
brush 

No  impact 

Lidar:  no  impact 
Orthophotos:  no  impact 

Medium  brush  and 
some  trees 

No  impact 

Lidar:  some  degradation  in 
surface  models  in  brush,  no 
impact  in  light  tree  cover 
Orthophotos:  will  not  show 
areas  beneath  trees  or 
brush 

Medium  to  heavy  trees 

No  impact 

Lidar:  surface  models  show 
some  degradation  in  heavy 
trees 

Orthophotos:  limited 
usefulness  in  areas  of 
medium  to  heavy  trees 

Full  canopy 

No  impact 

Lidar:  surface  models  show 
severe  degradation 
Orthophotos:  limited 
usefulness 

Terrain 

Rugged  (steep,  rocky) 

No  impact 

Lidar:  positional  accuracy 
will  be  somewhat  lower  in 
steep,  rough  terrain 
Orthophotos:  very  steep 
terrain  can  cause 
shadowing,  plan  data 
collection  for  hours  when 
the  sun  is  directly  overhead 

Rolling  (hills,  ravines, 
ruts) 

Level  (wide,  open,  flat) 

Surface 

clutter 

Uncluttered  through 
heavily  cluttered 

No  impact 

No  impact  from  surface 
clutter 

Geology 

Localized  or  regional 
ferrous  rocks  or  soil 

No  impact 

No  impacts  from  ferrous 
rocks  or  soil 

Survey  site 
shape 

Rectangular  vs  linear 
or  irregular 

Linear  or  irregular  shapes 
may  require  additional 
flight  lines 

Lidar:  no  impact 
Orthophotos:  no  impact 

5.2  Cost  Strategies 

There  are  several  strategies  for  cost  management  in  lidar  and  orthophoto  surveys: 

■  Occasionally,  projects  may  be  scheduled  to  share  mobilization  costs  between 
completely  unrelated  projects.  It  is  worthwhile  to  stay  in  touch  with  vendors  who 
may  be  working  in  the  same  area  to  see  if  projects  can  be  combined. 
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■  All  needed  areas  should  be  acquired  at  once.  Lidar  and  orthophotos  are  less 
expensive  per  unit  area  to  acquire  if  collected  in  larger  areas,  where  mobilization 
costs  may  be  shared  over  more  area  and  multiple  mobilizations  can  be  avoided. 

■  Lidar  and  orthophotos  should  be  acquired  together  if  both  data  sets  will  be 
needed.  Acquiring  data  concurrently  can  achieve  both  cost  savings  and  better 
data  integration. 

■  Lidar  and  orthophotos  may  be  used  in  both  the  site  assessment  and  cleanup 
phases  of  site  project,  as  well  as  in  ongoing  site  management.  As  such,  it  may 
be  possible  to  share  the  cost  of  the  survey  between  different  programs  or  funding 
sources. 

There  are  some  aspects  of  a  project  where  it  is  not  appropriate  to  economize: 

■  Data  calibration,  data  inspection,  and  other  data  processing  steps. 

Automated  algorithms  can  perform  most  (but  not  all)  data  processing  steps; 
however,  review  by  experienced  operators  is  essential  to  insure  data  quality. 
Vendors  should  be  asked  to  describe  their  calibration,  integration,  point 
classification  and  orthophoto  creation  process  in  their  proposals,  and  extremely 
low  bids  should  be  scrutinized  with  care  to  be  sure  that  process  steps  by 
experienced  operators  are  not  being  slighted. 

■  Adequate  survey  control.  Establishing  sufficient  survey  control  is  relatively 
inexpensive  and  provides  the  essential  means  of  data  adjustment  and  verification 
of  data  positional  accuracy.  Survey  control  should  not  be  minimized  except 
where  field  work  poses  a  safety  risk  to  the  crew. 

■  QA/QC  review  in  the  field.  Field  QA/QC  review  is  important  to  avoid  the  risk  of 
having  to  re-mobilize  and  re-acquire  data  in  the  event  that  bad  data  is  collected. 
This  is  true  even  if  the  cost  of  correcting  poor  quality  data  falls  to  the  vendor, 
since  re-acquisition  after  demobilization  can  causes  significant  delay.  The 
schedule  should  always  include  time  for  in-field  QA/QC  review  and  re-acquisition 
of  any  missed  or  erroneous  areas  while  the  crew  is  still  in  the  field. 

5.3  Additional  Contracting  Considerations 

The  lidar/orthophoto  industry  is  in  some  flux  at  the  time  of  this  report,  with  some  firms 
consolidating  and  new  firms  emerging.  The  level  of  experience  among  vendors  varies 
substantially,  with  some  firms  having  more  or  less  experience,  or  experience  only  in  a 
particular  market,  such  as  flood  plain  mapping,  the  electric  power  industry,  or  other 
applications.  Vendors  with  no  experience  with  munitions  sites  should  not  necessarily  be 
excluded  from  consideration,  since  most  vendors  use  equipment  that  will  perform  well 
for  a  variety  of  projects.  However,  the  wide  variety  of  experience  makes  it  important  to 
work  closely  with  all  vendors  to  make  sure  that  they  understand  the  unique  requirements 
of  munitions  investigations. 

A  discussion  of  vendor  qualifications  should  include  examples  of  the  vendor’s 
performance  when  data  collection  or  processing  did  not  go  as  planned.  For  instance, 
the  vendor  should,  if  possible,  provide  customer  contacts  as  to  its  willingness  to  re¬ 
acquire  or  re-process  areas  of  bad  data  in  the  event  of  equipment  problems  or  failure  to 
meet  project  specifications  (Appendix  A). 
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Lidar  vendors  are  often  booked  far  in  advance,  and  may  need  several  months  of 
advance  notice  to  plan  a  survey.  Therefore  the  general  rule  is  to  contact  vendors  as 
early  in  the  planning  process  as  possible  to  reserve  time.  Occasionally,  vendors  will  be 
able  to  accommodate  projects  on  short  notice  if  they  have  gaps  in  their  schedules  or  if 
other  projects  are  cancelled. 

It  is  appropriate  to  present  contract  specifications  to  several  vendors  and  request  cost 
and  technical  proposals.  Generally,  vendors  should  be  requested  to  describe  their 
technical  approach,  and  to  invite  the  vendor  to  present  alternative  approaches  that  the 
vendor  feels  will  meet  the  project  needs  better. 

In  addition  to  the  example  data  specifications  discussed  in  Appendix  B,  the  following 
additional  contract  provisions  should  be  considered: 

■  Contingencies  for  weather  delays.  In  coastal  areas  especially,  weather  can 
cause  severe  delays.  Vendors  should  be  asked  about  their  contingencies  for  bad 
weather,  and  whether  there  will  be  an  extra  charge  for  weather  delays  once 
mobilization  has  occurred. 

■  Replacement  of  poor  quality  data.  Typically,  lidar  and  orthophoto  acquisition 
contracts  specify  that  the  vendor  will  replace  out-of-specification  data  at  the 
vendor’s  cost  with  data  that  meets  the  contract  specifications.  Major  errors  that 
cause  the  data  not  to  meet  the  contract  specifications  can  result  in  substantial  re¬ 
processing  or  even  re-acquisition.  Minor  errors  include  delivery  of  occasional 
corrupt  data  files  or  data  blocks  in  an  incorrect  projection  and  datum. 

Given  the  amount  of  data  involved  in  a  large  lidar  and  orthophoto  survey,  delivery 
of  an  occasional  bad  file  is  tolerable  and  probably  inevitable.  Errors  of  this  type 
are  acceptable  as  long  as  they  are  infrequent  and  data  is  promptly  replaced. 
Nevertheless,  it  may  be  reasonable  to  stipulate  that  after  delivery  of  an  agreed- 
upon  number  of  poor  quality  data  files,  the  vendor  will  be  penalized. 
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7  GLOSSARY 

accuracy:  The  closeness  of  an  estimated  value  to  a  standard  or  accepted  value  of  a 
particular  quantity. 

anomaly:  A  geophysical  signal  above  geological  background  from  a  detected 
subsurface  object. 

artifacts:  In  lidar,  detectable  surface  remnants  of  buildings,  trees,  towers,  telephone 
poles  or  other  elevated  features  in  a  bare-earth  elevation  model.  Also,  detectable 
artificial  anomalies  that  are  introduced  to  a  surface  model  via  system-specific  collections 
or  processing  techniques. 

bathymetry:  The  measurement  and  study  of  water  depths. 

calibration:  The  process  of  identifying  and  correcting  for  systematic  errors  in  hardware, 
software,  or  procedures. 

conceptual  site  model  (CSM):  A  description  of  site  conditions  that  coveys  what  is 
known  or  suspected  about  the  sources,  releases  and  release  mechanisms,  contaminant 
fate  and  transport,  exposure  pathways,  potential  receptors,  and  risks. 

contours:  Lines  of  equal  elevation  on  a  surface.  An  imaginary  line  on  the  ground,  all 
points  of  which  are  at  the  same  elevation  above  or  below  a  specified  reference  surface. 

digital  elevation  model  (DEM):  A  generic  term  for  digital  topographic  and/or 
bathymetric  data  in  all  its  various  forms,  but  most  often  bare  earth  elevations  at  regularly 
spaced  intervals  in  x  and  y  directions.  Regularly  spaced  elevation  data  are  easily  and 
efficiently  processed  in  a  variety  of  computer  uses. 

digital  terrain  model  (DTM):  Similar  to  DEMs,  but  they  may  incorporate  the  elevation  of 
significant  topographic  features  on  the  land  and  mass  points  and  break  lines  that  are 
irregularly  spaced  to  better  characterize  the  true  shape  of  the  bare  earth  terrain. 

digital  surface  model  (DSM):  Similar  to  DEMs  or  DTMs,  except  they  may  depict  the 
elevations  of  the  top  surfaces  of  buildings,  trees,  towers,  and  other  features  elevated 
above  the  bare  earth. 

electromagnetic  induction  (EMI):  The  physical  process  by  which  a  secondary 
electromagnetic  field  is  induced  in  an  object  by  a  primary  electromagnetic  field  source. 

Geographic  Information  System  (GIS):  A  system  of  spatially  referenced  information, 
including  computer  programs  that  store,  manipulate,  analyze,  and  display  spatial  data. 

Global  Positioning  System  (GPS):  Technology  that  computes  the  three-dimensional 
position  of  an  object  in  space,  for  example  the  lidar  sensor,  using  signals  from  at  least 
four  orbiting  navigation  satellites. 

hillshade:  A  function  used  to  create  an  illuminated  representation  of  a  surface,  using  a 
hypothetical  light  source,  to  enhance  visualization  effects. 

horizontal  accuracy:  The  positional  accuracy  of  a  dataset  with  respect  to  a  horizontal 
datum. 

Inertial  Measurement  Unit  (IMU):  Technology  that  uses  gyroscopes  and 
accelerometers  to  compute  the  roll,  pitch,  and  heading  of  a  moving  object,  for  example  a 
lidar  sensor. 
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Inertial  Navigation  System  (INS):  A  navigation  aid  that  uses  a  computer  and  motion 
sensors  (the  IMU)  continuously  calculate  via  dead  reckoning  the  position,  orientation, 
and  velocity  of  a  moving  object  without  the  need  for  external  references. 

light  detection  and  ranging  (lidar):  An  instrument  system  that  measures  distance  to  a 
reflecting  object  by  emitting  timed  pulses  of  laser  light  and  measuring  the  time  between 
emission  and  reception  of  reflected  pulses.  The  measured  time  interval  is  converted  to 
distance. 

magnetometry:  The  technique  of  measuring  and  mapping  patterns  of  the  earth’s 
magnetic  field  as  modified  by  geology  or  ferrous  objects. 

nanometer:  A  unit  of  length  equal  to  one  billionth  of  a  meter. 

ordnance:  Weapons  of  all  kinds. 

orthophotograph:  A  digital  aerial  photograph  that  has  been  geometrically  corrected  for 
topographic  relief,  lens  distortion,  and  camera  tilt. 

orthorectification:  The  process  by  which  the  geometric  distortions  of  an  image  are 
modeled  and  accounted  for. 

positional  accuracy:  The  accuracy  of  the  position  of  features,  including  horizontal 
and/or  vertical  positions. 

redundant  array  of  independent  disks”  (RAID)  device:  A  device  containing  multiple 
hard  drives,  allowing  computer  users  to  achieve  higher  levels  of  storage  reliability  from 
low-cost  and  less  reliable  PC-class  disk-drive  components,  via  the  technique  of 
arranging  the  devices  into  arrays  for  redundancy. 

root  mean  square  error  (RMSE):  An  accuracy  assessment  for  measured  data  (e.g., 
lidar)  calculated  by  taking  the  square  root  of  the  average  of  the  set  of  squared 
differences  between  dataset  values  (i.e.,  lidar  derived  elevations  versus  field  surveyed 
elevations). 

triangulated  irregular  network  (TIN):  A  set  of  adjacent,  non-overlapping  triangles 
computed  from  irregularly  spaced  points  with  x/y  coordinates  and  z-values. 

unexploded  ordnance:  Military  munitions  that  have  been  primed,  fused,  armed,  or 
otherwise  prepared  for  action  and  that  have  been  fired,  dropped,  launched,  or  placed  in 
a  manner  constituting  a  hazard  to  operations,  installations,  or  personnel;  which  remain 
unexploded. 

vegetation  removal:  In  lidar,  the  correction  of  reflective  surface  elevations  so  as  to 
depict  the  elevation  of  the  bare  earth  terrain  beneath  the  vegetation. 

vertical  accuracy:  The  measure  of  the  positional  accuracy  of  a  dataset  with  respect  to 
a  specified  vertical  datum. 

wide  area  assessment:  Rapid  assessment  of  large  tracts  of  potentially  contaminated 
land  to  identify  those  areas  with  concentrated  military  munitions  that  require  detailed 
characterization. 
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Contract  specifications  should  be  reviewed  to  be  sure  that  they  address  the  following 
topics: 

1 .  Proposal  Due  Date 

2.  General  Project  Description 

3.  Project  Schedule  Requirements 

4.  Project  Specifications 

-  Project  size 

-  Project  area  general  location 

-  Project  area  boundaries 

-  Project  area  configuration:  corridor  versus  area,  general  size,  number  and 
size  of  separate  areas 

-  Requirements  related  to  on-site  logistics,  access  and  project  coordination 

5.  Lidar  Specifications 

-  Lidar  data  density  required  (overall  and  within  smaller  areas  as  appropriate) 

-  Horizontal  and  vertical  accuracy 

-  Lidar  flight  line  integration  specification 

6.  Field  QA/QC  Requirements 

-  Field  assessment  of  ground  data  density 

-  Field  assessment  of  data  overlap  and  coverage 

-  Criteria  for  scope  modifications/additions 

7.  Orthophoto  Specifications 

-  Orthophotos  required  or  not 

-  Orthophoto  pixel  size 

-  Orthophoto  to  lidar  alignment  specification 

8.  Survey  Control  Requirements 

9.  Data  Delivery  Projection  and  Datum 

10.  Deliverables 

-  Data  delivery  format 

-  Data  delivery  medium 

-  Supporting  files  including  block  index,  attribute  description,  mission  and 
QA/QC  report 

1 1 .  Proposed  Technical  Approach: 

-  Proposed  flight  platform  (helicopter  v  fixed  wing  aircraft) 

-  Sensor  description 

-  Laser  pulse  rate 
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Planned  flight  altitude 
Proposed  flight  line  overlap 
Proposed  control  point  network 
Point  classification  methods 
Other  relevant  details 
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The  following  represents  some  sample  language  for  a  request  for  proposal  for  a  lidar 
and  orthophoto  survey.  This  language  is  intended  as  a  starting  point,  and  should  be 
adapted  to  fit  the  needs  of  each  particular  project. 

Request  for  Proposals,  Lidar  and  Orthophoto  Survey  in  Support  of  UXO  Wide  Area 
Assessment  at  [SITE  NAME] 

Proposal  due  date:  close  of  business  [DATE], 

AGENCY  (Government)  is  soliciting  proposals  for  acquisition  of  lidar  and  orthophoto 
data  for  a  project  near  [GENERAL  LOCATION],  The  objective  of  the  lidar/orthophoto 
survey  is  to  support  site  assessment  and  cleanup  activities  related  to  the  historic  use  of 
the  site  as  a  military  bombing  range  [OR  OTHER  AS  APPROPRIATE],  A  further 
objective  is  to  support  ongoing  site  management.  Project  award  will  be  made  based  on 
technical  qualifications,  cost,  and  ability  to  meet  the  attached  delivery  schedule. 

1.  Project  Specifications 

1.1.  Project  site  size:  approximately  [XX]  acres,  approximately  [XX]  acres, 
approximately  [XX]  by  [YY]  kilometers. 

1 .2.  Project  area  location  and  project  boundaries:  Project  areas  are  shown  in 
attached  ESRI  shape  files. 

1 .3.  Lidar  data  density.  Lidar  point  density  specifications  are  designed  to 
detect  craters  and  other  objects  of  approximately  1  meter  in  diameter. 
Vendor  will  acquire  lidar  data  at  a  minimum  of  5  points/m2.  Lidar  point 
density  will  be  calculated  based  on  total  points  versus  total  area  for  each 
separate  area;  however  lidar  data  density  shall  not  fall  below  3  points/m2 
for  any  500  x  500  m  area. 

1 .4.  Lidar  data  specifications: 

-  Lidar  data  will  have  a  vertical  accuracy  of  +/- 15  cm  and  a  horizontal 
accuracy  of  +/-  60  cm  root  mean  squared  error,  compared  to  the 
established  survey  control  points. 

-  Flight  line  to  flight  line  calibration  error  for  lidar  points  will  not  exceed 
12  cm  root  mean  squared  error. 

-  Lidar  will  be  acquired  for  1 00%  of  the  site. 

-  Lidar  point  classification  will  be  sufficient  to  create  surface  models 
free  of  trees  and  large  vegetation,  and  all  small  vegetation  which  can 
be  distinguished  from  natural  ground  features. 

1 .5.  Orthophoto  data:  Digital  photography  will  be  acquired  for  this  project 
concurrently  with  lidar. 

-  Orthophotos  will  be  acquired  at  10  cm  pixel  size  [OR  OTHER  SIZE 
DEPENDING  ON  THE  SITE], 

-  Orthophotos  will  be  spatially  accurate  to  within  3  pixel  widths 
compared  to  surveyed  control  points. 

1 .6.  Control  and  calibration  structures.  Vendor  will  independently  occupy  a 
survey  control  point  at  the  airport  used  for  flight  operations.  Stationary  or 
kinematic  survey  control  points  shall  be  established  throughout  the 
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project  area  sufficient  to  demonstrate  the  required  vertical  and  horizontal 
accuracy  for  each  major  terrain  type  on  the  site.  The  vendor  shall  provide 
a  plan  for  survey  control  acquisition  for  approval  prior  to  mobilization. 

1 .7.  Delivery  projection  and  datum.  Vendor  will  deliver  lidar  in  [SPECIFY],  the 
same  projection  and  datum  as  the  attached  shape  files  showing  project 
area  boundaries. 

1 .8.  On-site  coordination.  Vendor  will  be  responsible  for  all  on-site  logistics. 
No  flight  access  constraints  are  known  for  the  site  [OR  SPECIFY 
KNOWN  SITE  CONSTRAINTS]. 

1 .9.  Project  coordination.  Vendor  will  coordinate  with  the  Government 
throughout  the  project.  Deviations  from  agreed  specifications  will  only  be 
undertaken  after  prior  consultation  with  the  Government,  which  will  be 
confirmed  in  writing  or  via  email.  The  Government  will  supply  an  on-call 
point  of  contact  for  consultation  during  field  acquisition  to  preclude  delays 
in  the  acquisition  program. 

1.10.  Field  QA/QC  Specifications.  Vendor  will  perform  QA/QC  checks  for 
complete  coverage,  data  density  and  estimated  spatial  accuracy  prior  to 
demobilization,  and  will  re-acquire  all  areas  that  are  missed  or  where  data 
appears  not  to  meet  the  specifications  stated  in  this  Scope  of  Work. 

1.11.  Intellectual  property.  The  Government  shall  have  unrestricted  rights  to  all 
delivered  reports  and  data.  The  Government  will  place  reports  and  lidar 
data  in  the  public  domain.  This  specification  shall  not  restrict  the  ability  of 
the  Vendor  to  resell  data  or  derivative  products. 

2.  Project  Schedule 

Data  collection  shall  begin  no  later  than  [DATE],  and  data  delivery  to  the 
Government  shall  be  complete  no  later  than  ten  weeks  following  the  end  of  data 
collection.  [DELIVERY  DATE  WILL  BE  ADJUSTED  FOR  SITE  SIZE]  However, 
earlier  data  collection  and  delivery  is  desirable,  and  vendor  selection  will  depend 
in  part  on  proposed  schedule  for  mobilization,  data  collection,  and  data  delivery. 

3.  Deliverables 

3.1 .  Lidar  data  format:  All  lidar  data  points  will  be  delivered  as  ASCII  comma- 
delimited  files  with  a  .csv  extension,  with  values  for  northing,  easting, 
elevation,  intensity,  flight  line,  gps  date  and  time,  and  classification  as 
ground  or  non-ground  return.  Lidar  point  data  will  be  delivered  in  blocks 
representing  1  square  kilometer,  or  other  tiling  system  as  mutually 
agreed.  Lidar  data  may  be  delivered  in  phases  as  mutually  agreed.  Data 
blocks  will  overlap  by  a  minimum  of  10  meters. 

3.2.  Orthophoto  data.  Orthophotos  will  be  delivered  in  .geotiff  format,  in 
blocks  no  larger  than  1GB  in  size,  or  other  tiling  arrangements  as 
mutually  agreed. 

3.3.  Block  index.  A  lidar  and  orthophoto  block  index  will  be  provided  in  ESRI 
shape  file  or  other  agreed-upon  format. 

3.4.  Delivery  medium:  lidar  point  data  will  be  delivered  via  external  hard  drive. 

3.5.  Acquisition  and  QA/QC  report.  Vendor  will  deliver  data  acquisition  and 
QA/QC  reports  sufficient  to  demonstrate  compliance  with  all 


2 


Guidance  Document:  Using  Lidar  and  Orthophotography 

in  UXO  Wide  Area  Assessment 
Project  Number  07E-MM2-012/MM-0737 

specifications  in  this  Scope  of  Work.  The  QA/QC  report  shall  be 
delivered  concurrently  with  the  last  shipment  of  lidar  and  orthophoto  data. 

4.  Proposed  Costs  and  Schedule 


Cost  Items:  (Costs  in  US  Dollars) 


Item 

Amount  ($) 

Mobilization/Demobilization 

Data  Acquisition 

Data  Processing 

Data  Analysis 

Total 

Proposed  mobilization  date: 

Proposed  delivery  date(s)  for  lidar  data: 


5.  Technical  Approach 

Please  briefly  describe  your  technical  approach  including: 

-  proposed  flight  platform  (helicopter  v  fixed  wing  aircraft)  and  reason  for 
selection 

-  make  and  type  of  sensor 

-  planned  laser  pulse  rate 

-  planned  flight  altitude 

-  flight  line  overlap 

-  conceptual  control  point  network 

-  proposed  point  classification  methods  and  how  these  will  be  optimized  to  the 
goals  of  the  survey 

-  suggested  technical  modifications  to  better  meet  the  goals  of  the  survey 
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APPENDIX  E 

FEATURE  DETECTION  AND  POINT  DECIMATION  -  ANALYSIS  RESULTS 


Appendix  E 

Feature  Detection  and  Point  Decimation  -  Analysis  Results 

In  this  table  the  plots  are  ordered  by  feature  size. 


Decimination  Level 


Feature 

Size 

lm) 


Plot# 


Lidar 

ref# 


Average  Point  Density  (pts/m2) 


Veg 

% 


Notes 


Lowest 

level 

visible 


Average  pt  density 


The  feature  is  very  faint  throughout  and  gone  by  4. 


The  feature  is  largely  gone  by  3,  but  because  it  is  surrounded  by  a 
cleaner  set  of  features,  it  could  probably  still  be  identified.  The  feature 
is  completely  gone  in  4  and  5. 


The  feature  is  clear  throughout 


11.4 


The  feature  in  1  through  3  is  difficult  to  distinguish  from  background 
noise.  At  4  and  5,  the  feature  is  completely  gone. 


55.7 


The  feature  is  discernable  in  1  and  2,  and  gone  by  3. 


34.2 


The  feature  remains  distinguishable  as  such  until  it  disappears 
completely  in  5. 


34.1 


The  feature  is  relatively  clear  through  3  and  could  possibly  be  picked 
out  in  4,  but  has  degraded  enough  in  5  to  make  it  invisible. 


35 


The  feature  is  fairly  clear  through  3  but  disappears  in  4  and  5 


33.7 


The  feature  is  largely  gone  by  3,  but  because  it  is  surrounded  by  a 
cleaner  set  of  features,  it  could  probably  still  be  identified.  The  feature 
is  completely  gone  in  4  and  5. 


The  feature  is  clear  through  3,  but  disppears  completely  in  4. 


42.1 
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3.3 


3.5 


3.7 


3.8 


3.8 


3.8 


3.8 


3.9 


4.0 


4.2 


4.3 


4.3 


4.3 


20 


22 


23 


24 


25 


26 


27 


29 


30 


44 


5924 


5738 


5512 


5463 


5466 


5580 


5719 


5465 


5740 


4088 


5615 


4077 


5350 
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28.7 

The  feature  is  clear  throughout 

5 

57.7 

The  feature  is  fairly  clear  through  4  but  is  gone  in  5 

4 

!  45.4 

The  feature  looks  reasonably  clear  in  1  and  2  but  has  degraded  enough 
in  3  to  look  more  like  surface  texture.  It  is  gone  from  4  and  5. 

2 

48.3 

The  feature  is  very  clear  through  4  but  disappears  completely  in  5. 

4 

36.3 

The  feature  seems  reasonably  clear  in  1  and  2  but  looks  like  surface 
texture  in  3  and  has  completely  disappeared  in  4. 

3 

43.8 

The  feature  is  clear  through  4  but  has  essentially  disappeared  in  5. 

5 

61.5 

The  feature  is  clear  in  1  and  2,  fairly  clear  in  3  and  4,  but  badly 
degraded  in  5 

4 

38.6 

The  feature  looks  relatively  clear  through  3  but  has  degraded  enough  in 

4  to  possibly  be  mistake  for  background  noise.  It  has  completely 
disappeared  in  5. 

4 

39 

The  feature  is  clear  through  3  but  fades  somewhat  into  the  surface 
texture  in  4  and  5. 

5 

46.3 

The  feature  looks  like  something  in  1  and  2.  By  3,  feature  looks  like 
background  noise,  and  is  gone  completely  from  4  and  5. 

3 

52.9 

The  feature  is  reasonably  clear  throughout 

5 

73.8 

The  feature  continues  to  be  visible  until  it  disappears  completely  in  5. 

4 

67 

Even  though  the  feature  changes  shape  considerably  by  5,  it  still  looks 
like  something  worth  investigating. 

5 

38.5 


The  feature  is  clear  throughout 


The  feature  is  clear  throughout 


63.4 


The  feature  is  more  visible  on  the  ortho  that  it  does  in  any  of  the  surface 
models.  It  might  be  noticeable  through  3,  but  not  in  4  or  5. 


47.8 


The  feature  changes  shape  entirely  in  3,  but  still  might  be  considered  to 
be  something.  It  has  degraded  further  in  4  and  gone  in  5. 


47.7 


The  feature  is  reasonably  clear  in  1  and  2  but  has  essentially 
disappeared  in  3,  4,  and  5. 


50.2 


The  feature  is  clear  through  3  but  is  pretty  much  gone  in  4  and  5. 


42.7 


The  feature  is  clear  through  3  but  is  beginning  to  look  like  normal 
surface  texture  in  4  and  5. 


53.5 


The  feature  survives  the  decimination  process  but  changes  shape 
considerably.  The  feature  is  slightly  visible  in  the  ortho,  but  probably 
only  because  there  is  a  white  circle  around.  It. 


43.2 


The  feature  is  fairly  clear  through  3  but  has  degraded  in  surface  texture 
by  4  and  5 


38.9 


Even  though  the  feature  fades  somewhat,  it  is  still  discernable  through 
level  4. 


53 


The  feature  is  relatively  clear  through  3,  but  fades  into  the  surface 
texture  in  4  and  5. 


51.7 


The  feature  is  clear  through  3,  fairly  clear  in  4,  but  badly  degraded  in  5. 


54.9 


The  feature  is  clear  throughout. 


37.4 
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5.8 


6.2 


6.5 


7.1 


8.0 


42 


43 


45 


46 


47 


5717 


5613 


5678 


3975 


5609 


47 


51.1 


47.5 


44.6 


62.8 


The  feature  is  clear  throughout 


The  feature  is  clear  through  4,  but  disappears  in  5. 


The  feature  remains  clear  throughout  though  it  changes  shape 
considerably  by  5. 


The  feature  has  changed  shaped  considerably  in  3,  but  still  appears  to 
be  a  hole  in  the  ground.  Not  visible  in  4  or  5. 


The  feature  is  large  enough  to  reamin  clear  throughout 


8.2 


49 


4029 


The  feature  doesn’t  entirely  disappear  but  by  3  it  has  started  looking  like 
a  natural  surface  feature. 


51.2 


8.2 


48 


5921 


The  feature  remains  clear  through  3,  but  has  degraded  badly  in  4  and  5. 


47 


8.6 


8.7 


8.8 


50 


51 


52 


10.3 


53 


10.4 


54 


11.3 


55 


5735 


5679 


4030 


5739 


5913 


5923 


The  feature  is  clear  throughout 


51.5 


The  surface  feature,  such  as  it  is,  remains  discernable  through  3  and 
somewhat  so  in  4,  but  all  but  disappears  by  5. 


55.5 


The  feature  contiunes  to  be  something  worth  noting  until  5  where  it  is  no 
longer  visible. 


41 


Feature  is  clear  throughout. 


46.9 


The  feature  looks  like  it  may  be  something  noteworthy  in  1  and  2,  is  less 
noticable  in  3  and  4,  and  has  disappeared  in  5. 


60.2 


The  feature  is  clear  throughout 


36.5 
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12.2 

56 

5583 

14.3 

57 

4035 

59 

4068 
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The  feature  is  reasonably  clear  throughout 

43.2 

5 

The  feature  is  visible  throughout  though  quality  decreases. 

46 

5 

63.3 

The  faint  dimple  of  a  feature  disappears  completely  in  2  and  is  never 
seen  again 

1 

Plot  #  1 


Feature  size:  3.2  m 
Vegetation  Density:  0  % 

Lidar  block  reference 
number:  3977 


Decimation  Levels: 

1:  13.8  pts/m2  (original) 
2:  6.9  pts/m2 
3:  3.4  pts/m2 
4:  1.7  pts/m2 
5:  0.8  pts/m2 
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Plot  #  2 


Feature  size:  1.9  m 
Vegetation  Density:  0  % 
Lidar  block  reference 
number:  4004 


Appendix  E:  Feature  Detection  and  Point  Decimation  -  Analysis  Results 
Page  7  of  64 


Plot  #  3 


Feature  size:  4.2  m 
Vegetation  Density:  46.4 

% 

Lidar  block  reference 
number:  4088 
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Plot  #  4 
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Plot  #  5 


Feature  size:  4  m 
Vegetation  Density:  47.44 

% 

Lidar  block  reference 
number:  4065 
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Plot  #  6 


Feature  size:  4.6  m 
Vegetation  Density:  47.76 

% 

Lidar  block  reference 
number:  4086 
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Plot  #  7 


Feature  size:  3.8  m 
Vegetation  Density:  36.31 

% 

Lidar  block  reference 
number:  5466 
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Plot  #  8 


Feature  size:  3.8  m 
Vegetation  Density:  43.82 

% 

Lidar  block  reference 
number:  5580 
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Plot  #  9 


Feature  size:  4.8  m 
Vegetation  Density: 
42.75  % 

Lidar  block  reference 
number:  5743 
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Plot  #  10 


Feature  size:  no  feature 
visible 

Vegetation  Density:  48.74 

% 

Lidar  block  reference 
number:  4078 
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Plot  #11 


Feature  size:  no  feature 
visible 

Vegetation  Density:  30  % 
Lidar  block  reference 
number:  5645 
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Plot  #  12 


Feature  size:  1.5  m 
Vegetation  Density:  50.78 

% 

Lidar  block  reference 
number:  5822 


1 


80  40  0 


Meters 
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Plot  #  13 


Feature  size:  2.6  m 
Vegetation  Density:  0  % 
Lidar  block  reference 
number:  3978 
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Plot  #  14 


t  9 


Feature  size:  2.7  m 
Vegetation  Density:  1 1.36 

% 

Lidar  block  reference 
number:  3991 
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Plot  #  15 


Feature  size:  2.9  m  wide 
3.7  m  long 

Vegetation  Density:  53.25 

% 

Lidar  block  reference 
number:  5351 
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Plot  #  16 


Feature  Size:  2.9  meters 
Vegetation  Density: 
55.7% 

Lidar  block  reference 
number:  3953 
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Plot  #  17 


Feature  size:  3.1  m 
Vegetation  Density:  42.46 

% 

Lidar  block  reference 
number:  5823 
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Plot  #  18 


Feature  size:  3.1  m 
Vegetation  Density:  33.72 

% 

Lidar  block  reference 
number:  5718 
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Plot  #  19 


Feature  size:  3.1  m 
Vegetation  Density:  34.95 

% 

Lidar  block  reference 
number:  5461 
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Plot  #  20 


Feature  size:  3.25  m 
Vegetation  Density:  28.67 

% 

Lidar  block  reference 
number:  5924 


3 


Meters 
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Plot  #  21 


Feature  size:  3.25  m 
Vegetation  Density:  42.06 

% 

Lidar  block  reference 
number:  4089 
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Plot  #  22 


Feature  size:  3.5  m 
Vegetation  Density:  57.7 

% 

Lidar  block  reference 
number:  5738 
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Plot  #  23 


Feature  size:  3.7  m 
Vegetation  Density:  45.43 

% 

Lidar  block  reference 
number:  5512 
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Plot  #  24 


Feature  size:  3.8  m 
Vegetation  Density:  48.33 

% 

Lidar  block  reference 
number:  5463 
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Plot  #  25 


Feature  size:  3.8  m  wide 
23.25  m  long 
Vegetation  Density:  61.5 

% 

Lidar  block  reference 
number:  5719 
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Plot  #  26 
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Plot  #  27 


Feature  size:  4  m 
Vegetation  Density:  38.97 

% 

Lidar  block  reference 
number:  5740 
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Plot  #  28 


Feature  size:  4.1  m 
Vegetation  Density:  57.2 

% 

Lidar  block  reference 
number:  5641 
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Plot  #  29 


Feature  size:  4.25  m 
Vegetation  Density:  52.9 

% 

Lidar  block  reference 
number:  5615 
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Plot  #  30 


Feature  size:  4.3  m 
Vegetation  Density:  73.79 

% 

Lidar  block  reference 
number:  4077 
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Plot  #  31 


Feature  size:  4.5  m 
Vegetation  Density:  38.5 

% 

Lidar  block  reference 
number:  4001 


Meters 
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Plot  #  32 


Feature  size:  4.5  m 
Vegetation  Density:  63.38 

% 

Lidar  block  reference 
number:  5468 
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Plot  #  33 
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Plot  #  34 


Feature  size:  4.8  m 
Vegetation  Density:  50.17 

% 

Lidar  block  reference 
number:  5521 
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Plot  #  35 


Feature  size:  5.1  m 
Vegetation  Density:  53.5 

% 

Lidar  block  reference 
number:  5569 
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Plot  #  36 


Feature  size:  5.2  m 
Vegetation  Density:  38.94 

% 

Lidar  block  reference 
number:  5736 


80  40  0 


Meters 
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Plot  #  37 


Feature  size:  5.2  m  wide 
10  m  long 

Vegetation  Density:  43.17 

% 

Lidar  block  reference 
number:  3965 
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Plot  #  38 


Feature  size:  5.3  m  wide 
Vegetation  Density:  52.97 

% 

Lidar  block  reference 
number:  5644 
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Plot  #  39 


Feature  size:  5.3  m 
Vegetation  Density:  51.68 

% 

Lidar  block  reference 
number:  5675 
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Plot  #  40 


Feature  size:  5.6  m  wide 
32.1  m 
14.6  m 

Vegetation  Density:  54.89 

% 

Lidar  block  reference 
number:  5715 
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Plot  #  41 


Feature  size:  5.8  m 
Vegetation  Density:  37.38 

% 

Lidar  block  reference 
number:  4033 
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Plot  #  42 


Feature  size:  5.8  m 
Vegetation  Density:  47.04 

% 

Lidar  block  reference 
number:  5717 
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Plot  #  43 


Feature  size:  6.2  m 
Vegetation  Density:  51.12 

% 

Lidar  block  reference 
number:  5613 
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Plot  #  44 


Feature  size:  6.3  m 
4.3  m 

Vegetation  Density:  67.05 

% 

Lidar  block  reference 
number:  5350 
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Plot  #  45 


Feature  size:  12  m 
6.5  m 

Vegetation  Density:  47.48 

% 

Lidar  block  reference 
number:  5678 
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Plot  #  46 


Feature  size:  7.1  m 
Vegetation  Density:  44.58 

% 

Lidar  block  reference 
number:  3975 
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Plot  #  47 


Feature  size:  8  m  wide 
52  m  long 

Vegetation  Density:  62.79 

% 

Lidar  block  reference 
number:  5609 


Appendix  E:  Feature  Detection  and  Point  Decimation  -  Analysis  Results 
Page  52  of  64 


Plot  #  48 


Feature  size:  8.2  m 
Vegetation  Density:  46.96 

% 

Lidar  block  reference 
number:  5921 
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Plot  #  49 


Feature  size:  8.2  m 
Vegetation  Density:  51.24 

% 

Lidar  block  reference 
number:  4029 
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Plot  #  50 


Feature  size:  8.6  m 
Vegetation  Density:  51.46 

% 

Lidar  block  reference 
number:  5735 


Appendix  E:  Feature  Detection  and  Point  Decimation  -  Analysis  Results 
Page  55  of  64 


Plot  #  51 


Feature  size:  8.7  m  wide 
12.6  m  long 

Vegetation  Density:  55.53 

% 

Lidar  block  reference 
number:  5679 
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Plot  #  52 


Feature  size:  8.8  m 
Vegetation  Density:  40.96 

% 

Lidar  block  reference 
number:  4030 
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Plot  #  53 


Feature  size:  10.3  m 
Vegetation  Density:  46.88 

% 

Lidar  block  reference 
number:  5739 
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Plot  #  54 


Feature  size:  10.4  m 
Vegetation  Density:  60.17 

% 

Lidar  block  reference 
number:  5913 
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Plot  #  55 


Feature  size:  1 1.3  m 
Vegetation  Density:  36.49 

% 

Lidar  block  reference 
number:  5923 
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Plot  #  56 


Feature  size:  12.2  m 
Vegetation  Density:  43.21 

% 

Lidar  block  reference 
number:  5583 
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Plot  #  57 


Feature  size:  14.3  m  wide 
28.4  m  long 

Vegetation  Density:  46.03 

% 

Lidar  block  reference 
number:  4035 
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Plot  #  58 
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Plot  #  59 


Appendix  E:  Feature  Detection  and  Point  Decimation  -  Analysis  Results 
Page  64  of  64 


APPENDIX  F 

FEATURE  DETECTION  AND  POINT  DECIMATION  -  SURFACE  MODEL  RESULTS 


Plot  #  1 


Feature  size:  3.2  m 
Vegetation  density:  0% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  3977 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  yes 


Plot  #  1 


Feature  size:  3.2  m 
Vegetation  density:  0% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  3977 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  yes 
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Plot  #  2 


Feature  size:  1.9  m 
Vegetation  density:  0% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4004 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  2 


Feature  size:  1.9  m 
Vegetation  density:  0% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  4004 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 


Appendix  F:  Feature  Detection  and  Point  Decimation  -  Surface  Model  Results 
Page  3  of 37 


Plot  #  2 


Vendor  Classified 
LIDAR  Points  -  1  m 


Feature  size:  1.9  m 
Vegetation  density:  0% 
Surface  method:  IDW 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4004 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 

Interpolated  surface  is 
generally  similar  to  TIN- 
derived  surface. 


Plot  #  2 


Feature  size:  1.9  m 
Vegetation  density:  0% 
Surface  method:  IDW 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  4004 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  yes 

Interpolated  surface  is 
generally  similar  to  TIN- 
derived  surface. 


Vendor  Classified 
LIDAR  Points  -  1  ft 


f  I 


Appendix  F:  Feature  Detection  and  Point  Decimation  -  Surface  Model 
Page  4  of  37 


Reclassified  LIDAR 
Points  -  1  m 


o 

;\r  '■■'■■'fi 

80  40  0  80 

Meters 


Reclassified  LIDAR 
Points  -  1  ft 


i  »  •!  '  r*  • 

;  r  *  t*  'i  f  J  ‘ 

rP'j? rfl  •  iSv  * 


Plot  #  3 


Feature  size:  4.2  m 
Vegetation  density:  46.35% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4088 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 


Plot  #  3 


Feature  size:  4.2  m 
Vegetation  density:  46.35% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  4088 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 


Meters 
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Plot  #  3 


Feature  size:  4.2  m 
Vegetation  density:  46.35% 
Surface  method:  IDW 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4088 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 

Interpolated  surface  shows 
additional  features  than 
TIN-derived  surface. 


Plot  #  3 


Feature  size:  4.2  m 
Vegetation  density:  46.35% 
Surface  method:  IDW 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  4088 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 

Interpolated  surface  shows 
additional  features  than 
TIN-derived  surface. 
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Plot  #  4 


Feature  size:  2.9  m 
Vegetation  density:  34.16% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4007 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  4 


Feature  size:  2.9  m 
Vegetation  density:  34.16% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  4007 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Appendix  F:  Feature  Detection  and  Point  Decimation  -  Surface  Model  Results 
Page  7  of 37 


Plot  #  5 


Feature  size:  4  m 
Vegetation  density:  47.44% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4065 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  5 


Feature  size:  4  m 
Vegetation  density:  47.44% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  4065 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  6 


Feature  size:  4.6  m 
Vegetation  density:  47.76% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4086 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  6 


Feature  size:  4.6  m 
Vegetation  density:  47.76% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  4086 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  no 
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Plot  #  7 


Feature  size:  3.8  m 
Vegetation  density:  36.31% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5466 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  7 


Feature  size:  3.8  m 
Vegetation  density:  36.31% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  5466 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  8 


Feature  size:  3.8  m 
Vegetation  density:  43.82% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5580 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 


Vendor  Classified 
LIDAR  Points  -  1  m 


Plot  #  8 


Feature  size:  3.8  m 
Vegetation  density:  43.82% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  5580 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  9 


Feature  size:  4.8  m 
Vegetation  density:  42.75% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5743 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  9 


Feature  size:  4.8  m 
Vegetation  density:  42.75% 
Surface  method:  TIN 
DEM  cell  size:  0.3  m 
Lidar  block  reference 
number:  5743 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Meters 
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Plot  #  10 


Feature  size:  no  feature 
visible 

Vegetation  density:  48.74% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4078 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  11 


Feature  size:  no  feature 
visible 

Vegetation  density:  30.0% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5645 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  12 


Feature  size:  1.5  m 
Vegetation  density:  50.78% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5822 


Vendor  Classified 
LIDAR  Points 


Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  13 


Feature  size:  2.6  m 
Vegetation  density:  0% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  3978 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  yes 
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Plot  #  14 


Feature  size:  2.7  m 
Vegetation  density:  1 1.36% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  3991 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  yes 


Plot  #  15 


Feature  size:  2.9  m  wide, 
3.7  m  long 

Vegetation  density:  53.25% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5351 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  16 


Feature  size:  2.9  meters 
Vegetation  density:  55.7% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  3953 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  yes 


Plot  #  17 


Feature  size:  3.1  m 
Vegetation  density:  42.46% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5823 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  18 


Feature  size:  3.1  m 
Vegetation  density:  33.72% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5718 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 


Plot  #  19 


Feature  size:  3.1  m 
Vegetation  density:  34.95% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5461 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  20 


Feature  size:  3.25  m 
Vegetation  density:  28.67% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5924 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  21 


Feature  size:  3.25  m 
Vegetation  density:  42.06% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4089 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  22 


Feature  size:  3.5  m 
Vegetation  density:  57.70% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5738 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  23 


Feature  size:  3.7  m 
Vegetation  density:  45.43% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5512 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  24 


Feature  size:  3.8  m 
Vegetation  density:  48.33% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5463 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 


Aerial  Photo 


Plot  #  25 


Feature  size:  3.8  m  wide, 
23.2  m  long 

Vegetation  density:  61.50% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5719 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  26 


Feature  size:  3.9  m 
Vegetation  density:  38.58% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5465 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  27 


Feature  size:  4.0  m 
Vegetation  density:  38.97% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5740 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  28 


Feature  size:  4.1  m 
Vegetation  density:  57.20% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5641 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  29 


Feature  size:  4.25  m 
Vegetation  density:  52.90% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5615 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Meters 
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Plot  #  30 


Feature  size:  4.3  m 
Vegetation  density:  73.79% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4077 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  31 


Feature  size:  4.5  m 
Vegetation  density:  48.85% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4001 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  no 


Appendix  F:  Feature  Detection  and  Point  Decimation  -  Surface  Model  Results 
Page  23  of  37 


Plot  #  32 


Feature  size:  4.5  m 
Vegetation  density:  63.38% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5468 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  33 


Feature  size:  4.7  m 
Vegetation  density:  47.67% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4014 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  34 


Feature  size:  4.8  m 
Vegetation  density:  50.17% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5521 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  35 


Feature  size:  5.1  m 
Vegetation  density:  53.50% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5569 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  36 


Feature  size:  5.2  m 
Vegetation  density:  38.94% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5736 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  37 


Feature  size:  5.2  m  wide 
10.0  m  long 

Vegetation  density:  43.17 

% 

Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  3965 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  yes 
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Plot  #  38 


Feature  size:  5.3  m 
Vegetation  density:  52.97% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5644 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  39 


Feature  size:  5.3  m 
Vegetation  density:  51.68% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5675 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  40 


Feature  size:  5.6  m  wide, 
32.1  m  long,  14.6  m  long 
Vegetation  density:  58.89% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5715 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 


Plot  #  41 


Feature  size:  5.8  m 
Vegetation  density:  37.38% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4033 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  42 


Feature  size:  5.8  m 
Vegetation  density:  47.04% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5717 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 


Plot  #  43 


Feature  size:  6.2  m 
Vegetation  density:  51.12% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5613 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  44 


Feature  size:  6.3  m,  4.3  m 
Vegetation  density:  67.05% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5350 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  45 


Feature  size:  6.5  m,  12  m 
Vegetation  density:  47.48% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5678 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  46 


Feature  size:  7.1  m 
Vegetation  density:  44.58% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  3975 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  yes 


Plot  #  47 


Feature  size:  8.0  m 
Vegetation  density:  62.79% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5609 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  48 


Feature  size:  8.2  m 
Vegetation  density:  46.96% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5921 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  49 


Feature  size:  8.2  m 
Vegetation  density:  51.24% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4029 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  50 


Feature  size:  8.6  m 
Vegetation  density:  51.46% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5735 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  51 


Feature  size:  8.7  m  wide, 
12.6  m  long 

Vegetation  density:  55.53% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5679 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  52 


Feature  size:  8.8  m 
Vegetation  density:  40.96% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4030 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  no 


Plot  #  53 


Feature  size:  10.3  m 
Vegetation  density:  46.88% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5739 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  54 


Feature  size:  10.4  m 
Vegetation  density:  60.17% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5913 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  55 


Feature  size:  11.3  m 
Vegetation  density:  36.49% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5923 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  56 


Feature  size:  12.20  m 
Vegetation  density:  43.21% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  5583 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 


Plot  #  57 


Feature  size:  14.3  m  wide, 
28.4  m  long 

Vegetation  density:  46.03% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4035 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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Plot  #  58 


Feature  size:  17.4  m  wide, 
27.2  m  long 

Vegetation  density:  48.85% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  3987 

Additional  features  visible: 
possible 

Existing  features  more 
clearly  visible:  no 


Plot  #  59 


Feature  size:  3  m 
Vegetation  density:  63.28% 
Surface  method:  TIN 
DEM  cell  size:  1  m 
Lidar  block  reference 
number:  4068 

Additional  features  visible: 
no 

Existing  features  more 
clearly  visible:  possible 
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